Why marketers should be cautious using email open rate as success metric

Despite many warnings against basing an email campaign’s success or failure on its open rate, it remains the most popular measurement, according to a 2023 benchmark report from Mailjet by Sinch.

In the study, 42.7% of email marketers said open rate is one of their most important success metrics, closely followed by clickthrough rate (42.1%). In contrast, the most frequently cited business or objective-related metric was conversion rate (26%), while just 7.6% of marketers cited ROI as an important metric.

It is easy to understand why marketers might rely on some metrics more than others. Activity metrics are gathered and compiled in most email service provider (ESP) reports and are easily accessed, while strategic or business metrics require a close integration of all the platforms a marketer uses in an email programme and are seldom reported in a common dashboard. Further, marketers might not easily gain access to the key revenue or cross-channel engagement metrics that can shed even more light on programme performance.

The consequences of using the wrong success measurements can be significant. It can lead marketers to declare a campaign or email programme as having achieved its goal when it might actually be failing, or to undervalue a successful one.

What to remember when using email open rate

Marketers using open rate as a success metric should bear the following points in mind:

1. A true ‘open’ may not be registered as such

Email marketing software providers track open rates via a transparent image known as a tracking pixel.
.
‘Opens’ reflect the number of times this tracking pixel has loaded. This means that even if a person opens an email and reads its text, if images are not enabled, the email will not be recorded as opened. Many email clients, including Outlook and Gmail, disable images by default, while recipients who opt for plain text emails will also not receive the tracking pixel, which can only be included in HTML emails.

2. Some email clients may over count ‘opens’, thus inflating the results

In cases where images are enabled by default, an ‘open’ will be registered each time the person clicks the email, even if it only shows up in the preview pane. Some email clients, such as Hotmail, will automatically open email as the person scrolls through their inbox.
.
Note also that auto responses, such as out of office replies, may be counted as opens by some less advanced email marketing software.

3. Apple’s MPP artificially inflates open rates

Finally, the introduction of Apple’s Mail Privacy Protection (MPP) feature, which artificially inflates open rates to mask its users’ email activity, further undermines the open rate as an effective success measure, even for basic awareness campaigns.
.

The effects of MPP are still being measured. There is no single standard for the rate at which MPP distorts the actual open rate of a campaign because it depends on several factors:

• The percentage of subscribers who open and read email in Apple desktop or mobile Apple devices (including the iPhone and Apple Watch).
• How many have updated their devices’ operating systems to iOS 15, which introduced MPP.
• How many users have turned on the ‘Protect Mail Activity’ feature on their devices.

However, it is generally believed among multiple sources that track email client share of market (measured largely by open rates) that the MPP adoption level has stabilised enough for marketers to be able to gauge the impact on their engagement measures. According to email deliverability vendor Validity’s Guy Hanson: “80% of all [email] open events are MPP-generated false positives”.

One quick way to measure MPP’s impact is to compare the open rate from before 20 September 2021, when the feature was introduced, with open rates that have been recorded since that date. If other measures, including click and unsubscribe rates (but not click-to-open), remain stable while the open rate appears to be higher, the marketer can reasonably assume that MPP is driving a large measure of that increase.

Marketers recover from MPP metrics shakeup

Jess Evans-Tudor, Head of CRM, Rated People:

“When Apple’s Mail Privacy Protection [MPP] was first introduced in September 2021, it threw off our reporting like crazy. We had been trying to work out what a healthy metric would be to replace it. Now that we have annualised the data, we know it’s not super-accurate, but we can do a fair year-on-year comparison when we are looking at older metrics.

“We also had to go to generic numbers like clickthrough rate. Understanding the impacts of our subject lines for that first year was very hard. But when you use user engagement with the communications, [you can ask] ‘Are they actually clicking?’ and ‘What’s your unsubscribe rate?’

“Losing the open rate didn’t affect our automations or targeting because our segmentation at that point was more based on purchase rate and engagement, and actual clickthrough engagement.”

A business case for using open rate

Kelly Haggard, VP, Marketing Innovation, Synchrony:

“We’ve moved away from open rate, but there was still a business need for us. Some of our customer reports are based on predictive open rate, which we worked on with our ESP. Depending on what the business need is, there’s a report or a resource available to get there.”