One of Econsultancy’s main retention and sales channels is email marketing. Duh! Same as nearly every other company on the planet.

Email works because it’s the foundation of online marketing. But... is Econsultancy's subject line language optimal?

We analysed the emotional power of 82 of its recent email subject lines and here’s what we discovered.

The first challenge is to understand why some subject lines are good, and why some are bad.

Subject lines are important. This isn’t news. But how can you quantify the 'goodness' and 'badness' of a subject line?

Consider these two subject lines Econsultancy sent out earlier in 2015:

  1. Marketers, vent your frustrations here!
  2. Marketing Pain Points – share your stresses, get free report

These two subject lines invited people to fill out a survey about marketing pain points.

Before we look at results, here's some similarities and differences between the two subject lines:

  1. Version A is shorter than Version B.
  2. Version A uses an exclamation point, while Version B doesn’t have any ending punctuation.
  3. Version A begins with a collective noun, while Version B begins with a proper noun.
  4. Version A uses the word ‘frustrations’, while Version B uses ‘stresses’.
  5. Version A has a comma to separate fragments, while Version B uses a hyphen.
  6. Version B offers a tangible incentive, while Version A doesn’t.
  7. Version B uses the word ‘free’, while Version A doesn’t.
  8. Version B uses jargon (‘pain points’), while Version A is more colloquial.
  9. Both use the imperative voice.
  10. Both use second person pronouns.
  11. ….. and so on (I could go on – there are dozens of differences and similarities between the subject lines!)

Which subject line won – Version A or B?

Version B won with a 7% open rate lift (at 99% confidence). Whoo hoo!

So now the important question: why did it win, and what can Econsultancy learn from this?

As mentioned above, there’s a litany of ways that Versions A and B differ. Dozens, or maybe even hundreds.

Here are a few plausible hypotheses that this split test could answer:

  1. Longer subject lines are better than shorter.
  2. Using the word ‘free’ is good.
  3. Exclamation points are bad.

And so on. The list could go on ad infinitum.

This is just two subject lines! What happens when you look at dozens, hundreds or thousands of them?

How can you tell which are the best… and understand why they’re the best so you can replicate the success? 

What about these 10 subject lines?

These subject lines are the ones that experienced the biggest variation from the mean.

Subject Line StDevs from Mean
Upcoming training course I thought you'd be interested in 5.65
Where will marketers prioritise spending in 2015? -2.59
eCRM and analytics - join a three-day masterclass 1.79
Training: content and SEO working together -1.45
*drum roll* And the top 100 UK digital agencies are.... 1.34
Adopting a lean methodology: training in agile marketing 1.22
Understanding the Customer Journey - get your free report -1.20
What are marketers’ priorities in the year ahead? -1.12
Understanding the Customer Journey - new report 0.91
Intensive training in developing a positive customer experience -0.78

(Note: we normalised the results to determine the distance from the mean. This is to quantify variance that exists while controlling for other factors. A positive number means it was good, and a negative number means it was bad. The larger the number, the greater magnitude of its goodness or badness.)

Remember the differences we found between Versions A and B above? Let’s say we can find 10 differences between any two subject lines in the set of 10 above.

This means that A is different from B in 10 ways, from C in 10 ways, and D in 10 ways, plus B is different from C in 10 ways, D in 10 ways... and so on.

Suddenly you have a combinatorial explosion of millions of hypotheses, all of which could be correct. Discovering which is ‘the one’ is like finding the proverbial needle in the haystack.

So, what’s an email marketer to do? 

Here’s how you can quantify why some subject lines work and others don’t

You read a subject line as a chunk of text and emotionally respond to it. That’s how language works. You ingest an entire sentence, have a subconscious response, and act upon your impulse. Email subject lines work when they pique particular emotions in recipients.

But how do you quantify the emotions that resonate with your audience?

There are a couple of challenges here. First of all, hand-quantifying emotion is wrought with cognitive biases. And second, it’s a time-consuming and cumbersome task.

We ran Econsultancy’s data through our email subject line sentiment analysis tool - Phrasee Pheelings. It’s the world’s first technology that measures the emotional power of any of your previous subject lines.

Here’s the bad news for Econsultancy

Econsultancy tends towards two distinct emotional clusters in their subject lines, loosely described as:

  1. Direct and devoid of personality (i.e. ‘The Consumer Conversation - new report’) - in blue below
  2. Mid-level familiar and curious (i.e. ‘How healthy is your content strategy?’) - in pink below

Phrasee Pheelings 2 cluster

The response variance of the two clusters was noticeable, though small. Why is this?

When marketers have to write loads of subject lines in their day jobs, they either

  1. follow the status quo; or
  2. follow gut instinct.

Both of these strategies run many risks, not least of which are:

  • If the status quo subject lines weren't good in the first place, you’re just polishing a turd. You aren't progressively optimising your language decisions.
  • If unquantifiable assumptions (like the Versions A and B comparison above) drive your gut instinct, you are making uninformed decisions.

People tend to do the same things over and over, because “if it ain’t broke, don’t fix it.” The results don’t get worse, but also they don’t sustainably improve.

This is a missed opportunity.

Here’s the good news for Econsultancy

Econsultancy’s two main emotional clusters don’t vary much, and the results don’t either. But this is an opportunity in itself!

There are limitless possible emotional combinations of language available. Knowing which work best for your audience is impossible without testing.

The key is to get split testing. But don't test pointless things like changing a single word. That’s a tiny learning point that won’t tell you much about your audience. It will just make you buy a thesaurus.

Instead, test out big things. Like the overall emotional power of your subject lines.

And do it in a methodological, robust, quantifiable manner.

So, Econsultancy, here’s our advice to you:

When writing your subject lines, test the emotional impact of your subject lines.

You’ve tested out two emotional clusters ad infinitum, and the results aren’t bad. But what about the clusters you haven’t tried yet?

For example:

  • What about high urgency/’FOMO’ sentiments? There’s few like this in the sample set... but what if they're key drivers for your audience? In orange below.
  • How about more familiar tones? Econsultancy has a friendly brand voice, yet that doesn’t translate in your subject lines. In purple below.
  • And, what about high levels of curiosity and ambiguity? In green below.
  • And so on, and so on... there's limitless combinations to consider.

Phrasee Pheelings Untested clusters

To complicate matters – none of these emotions exist in isolation. You can have something that’s both urgent and familiar, for example. Or something that’s not that urgent, but has middling values of both familiarity and curiosity.

The challenges are:

  • to quantify what emotions to test;
  • to generate subject lines that adhere to these emotional clusters; and
  • to measure on an ongoing basis response, while controlling for external factors.

When testing out emotional clusters, some will work, and some won’t.

Your audience responds to the emotional power of your subject lines. Until you figure out what combination of emotions work the best, you’re using suboptimal subject lines.

The take-home point is this:

Small language sets are suboptimal if you don't quantify their success or failure. You don’t know if what you’re doing is good… or if you’re missing an opportunity.

Big thanks go to…

What I like about the good people at Econsultancy is they’re always up for trying new things… and sharing what they learn with their audience. They love innovation and that’s why they’re such a well-respected group.

So thanks a lot to the Econsultancy team for sharing their data with us here at Phrasee, and for being good sports through the process. 

And it’s not just us who think this is important…

When writing this blog post I received the following email from Econsultancy’s Festival of Marketing. Check out the subject line:

 

Emotional power is what makes marketing awesome.

We all know that emotion drives response. That’s not the opportunity.

The opportunity is to quantify emotion… then optimise based on the results… and then profit from better subject lines.

Parry Malm

Published 31 August, 2015 by Parry Malm

Parry Malm is the CEO of Phrasee and a contributor to Econsultancy. Connect with him on LinkedInTwitter or Google+.

25 more posts from this author

You might be interested in

Comments (12)

Comment
No-profile-pic
Save or Cancel
Avatar-blank-50x50

Len Diamond, Principal / Writer at Len Diamond Technical/Marketing Communications

You have proved conclusively that "Free" has strong appeal. This tells us that all the advertisers who used it over the last 70-80 years weren't just lucky.

over 2 years ago

Parry Malm

Parry Malm, CEO at Phrasee Ltd.

@Len thanks for the comment - but I think you must have been reading a different article.

over 2 years ago

Avatar-blank-50x50

Len Diamond, Principal / Writer at Len Diamond Technical/Marketing Communications

Here are a few plausible hypotheses that this split test could answer: Using the word ‘free’ is good.

over 2 years ago

Parry Malm

Parry Malm, CEO at Phrasee Ltd.

@Len which, of course, is one of many hypotheses, and is unproven - you'll also remember i noted how "status quo" and "gut instinct" are the natural enemies of optimal copywriting..

I would suggest that perhaps taking one line out of context isn't the best way to interpret the entirety of the post!

over 2 years ago

Avatar-blank-50x50

Adam Thomas Harper, Digital Sales and Marketing Assistant at Papertrail

@Parry Really interesting stuff! I'm looking forward to seeing more of this kind of research appearing in academic journals - I can definitely see e-mail marketing theory heading in this fresh direction :)

@Len I think the overall point of the article was to say that only through testing and optimisation can e-mail subject lines be improved (that which should be on a continuous basis anyway), and that Phrasee can help you to establish a solid framework from which to do so.

In any case, keep up the good work Mr Malm.

over 2 years ago

Victoria Prime

Victoria Prime, CRM Marketing Officer at A University

I love how in my most serious voice, to some of the 'big wigs' I blindly read out "If the status quo subject lines weren't good in the first place, you’re just polishing a turd..."

It even started off suggestions for subject lines such as 'we can't polish a turd, but we can roll it in glitter'. Maybe some Phrasee testing required for this one...

over 2 years ago

Parry Malm

Parry Malm, CEO at Phrasee Ltd.

@adam our chief scientist is a published PhD in AI. The research is there and we are commercially applying it. And btw thanks for the kind words!

@victoria glitter is great until it gets embedded in your carpet. Trust me, I've lost many a damage deposit due to errant glitter bombs! :)

over 2 years ago

Avatar-blank-50x50

Adam Thomas Harper, Digital Sales and Marketing Assistant at Papertrail

@Parry excellent, could you post a couple of links to his research papers? Thanks!

over 2 years ago

Victoria Prime

Victoria Prime, CRM Marketing Officer at A University

@Parry who hasn't? :)

over 2 years ago

Parry Malm

Parry Malm, CEO at Phrasee Ltd.

@Adam this research that has come out of Stanford is quite interesting looking at recursive deep learning to identify high-order interacations between words and phrases:
http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf

over 2 years ago

Pete Austin

Pete Austin, CINO at Fresh Relevance

@Parry That research provides a framework for assigning a sentiment value to an arbitrary piece of English text. It would be useful for e.g. assessing shoppers' opinions based on their written feedback, though they could fool it easily if they wanted to.

It does not seem useful in assessing audience reaction to subject lines, because a highly positive subject line - "this product is the best, it's wonderful, you must buy it, and such great value!" is unlikely to persuade shoppers of anything except to hit the spam button. And I don't see how the scheme could usefully be adapted.

about 2 years ago

Parry Malm

Parry Malm, CEO at Phrasee Ltd.

@Pete

You'll notice above that the results do direclty relate to response metrics. I can totally understand that it's a hard concept to grasp. But you'll note as well that the technology is specifically built for subject lines, so your point about an "arbitrary piece of English text" is incorrect.

about 2 years ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.