Emojis are in lots of subject lines these days. But do they work?

We get asked this question all the time. Here’s what we learned about emojis by looking at >1.5m subject lines.

Should we Emoji… or should we eNOji?

Ever since the Oxford English Dictionary named the “Laughing face with tears of joy” as the word of the year in 2015, marketers have been all about emojis. And so you see them everywhere, because just because you can do it… you should do it. Right?

Not so fast. Just because you can do something doesn’t mean you should. 

We know a little bit about email subject lines. In fact, our business at Phrasee is predicated on using AI to generate and optimise subject lines. So, our team of 30-odd people is solely dedicated to understanding the linguistic attributes of subject lines that work, and conversely, those that don’t. Also, we’re great at parties. 

Anyway, one day our linguists asked our data science team the age old question: should we Emoji… or should we eNOji? 

The answer was a resounding:

And so we started researching.

We started by looking at a huge volume of subject lines, in the hundreds of thousands, to first determine the frequency of subject line use. Overall, about 5% of global subject lines over the last 12 months include one or more emojis. So, not mega, but not insubstantial.

But, that doesn’t tell the whole story. Of the 5% of subject lines with emojis, about 1500 different emojis were used. That’s a lotta emojis! (Note: we included things like emoticons and whatnot in the sample set - so, any non-word-based linguistic constructs.)

Some trends became apparent just by eyeballing the frequency table. Here’s the top five:

Clearly, all you email marketers are stars who love holidays, amirite? 

At this point, we had shown that emoji usage had become quite widely adopted, presumably in the travel industry, and perhaps in the organ transplant industry as well. That’s great to know… but that wasn’t our question. 

So we went back to our data scientists, and asked a much more specific question:

Do emojis improve or decay performance of subject lines?

This question is more challenging to answer, as natural (i.e. unstructured) data doesn’t tell the whole story.

See, a subject line is made up of a multitude of components. To give you an idea of the scale of the linguistic complexity within a single subject line: our deep learning engine is trained to consider roughly 750,000 features when determining efficacy of a subject line. Emojis comprise some of those features, but there are MANY more. 

Further, deconstructing an advanced deep learning system is nigh on impossible. See, there’s a tradeoff between a model’s interpretability and accuracy. The less sophisticated the model, the more you can post-hoc explain it. As the model’s sophistication increases, the interpretability decreases at a disproportionately high rate.

(Tangent: if you ever see an “AI” model and people can easily deconstruct it to tell you why it made the choice it made… then chances are the system isn’t terribly sophisticated, and you probably just got ripped off).

Still, people kept asking us, “Should I be using emojis”? So our intrepid data detectives got to work.

Here’s how we designed the experiment.

To isolate - and measure - emojis as a (non) causal variable requires a specific experimental methodology.

The only other direct experiments we’ve seen in the past are single-subject-line tests, where you have A versus B - one with an emoji, and one without. This, however, doesn’t allow for repeatability. So it’s like those studies you see, where one day red meat will kill you, and the next day it’ll save your life. Pretty much not worth the paper it’s written on. 

So, our appetites for emoji research not yet sated, we partnered with one of our customers - a huge global brand - to do a proper experiment, following a proper scientific method.

Over the course of 14 campaigns, 10 randomly selected groups (about 50k in size each) received one of 10 different subject lines over 14 sends - five without emojis, and then the same five with emojis. This allowed us to test 70 different emojis across thousands of people.

Note: the goal of the experiment wasn’t to find a “good” emoji - it was to answer whether or not emojis improved or decayed response. That’s why we used a large variety of emojis.

We then tabulated the results, and made a fancy chart, because that’s what you do with data:

So, what did we learn?

Emojis are like Sex Panther cologne:

 

60% of the time, they work every time. When they work (about 60% of the time), they spike open rates by about a quarter of a standard deviation. When they don’t work (about 40% of the time), they decay response by about the same amount.

One interpretation of this result would be “inconclusivity”... but we weren’t satisfied with that. So we looked at the winning and losing clusters in more detail. And we learned something interesting.

Emojis are language amplifiers. 

An emoji, in itself, won’t make or break a subject line. The data proves this. But, they can be an additive - or subtractive - linguistic feature. 

What an emoji does is one of two things:

  1. It makes a bad subject line worse
  2. Or it makes a good subject line better

When emojis make bad subject lines worse…

If you’re already high-pressuring your customers into buying your stuff using misleading tactics, then adding in emojis will make them even spammier. Here’s an example of some subject lines that seem to be doing this:

sears

Spam in, spam out. An emoji ain’t gonna make these awful subject lines good. With trash like this, you may get a short-term spike in response... but it’ll drop off, I guarantee it. Tactics like this assume your customers are stupid, and that they’ll fall for it time and time again. But - this just in - your customers aren’t stupid. So you shouldn’t treat them so.  

This strategy smacks of desperation… with an emoji.  

When emojis make good subject lines better…

If your subject lines are already good… and your emojis are contextually relevant… then guess what? They make your end result even better. For example, Domino’s Pizza:

dominos email

Note how the emoji usage flows within the context of the subject lines, and they aren’t being over-used. The subject lines are on-brand, and are effective. And - not all of them contain an emoji, because not every subject line will benefit from one. 

This strategy smacks of delicious pizza… with an emoji. 

Wanna find out more? Sure, no probs.

Phrasee has put together a fancy report (Editor’s note: registration required) covering all of the above in way more detail, with a bunch more stats and findings.

Emojis are here to stay, so don’t ignore them.

But at the same time, don’t look at them with short-term goggles, and simply use them to become a better spammer. If you’re already a spammer, then go ahead, throw in some emojis. It’ll work once, maybe twice.

Your customers aren’t stupid, so don’t treat them as such.

Emojis are the biggest linguistic revolution since the QWERTY keyboard, and they are shaking things up and fundamentally changing how we communicate.

Unlike a picture, emojis don’t say a thousand words. 

But they can make the rest of your words better. So use them wisely.

More from Parry Malm:

More from Econsultancy:

Parry Malm

Published 22 November, 2017 by Parry Malm

Parry Malm is the CEO of Phrasee and a contributor to Econsultancy. Connect with him on LinkedInTwitter or Google+.

26 more posts from this author

You might be interested in

Comments (1)

Avatar-blank-50x50

Leon Klatt, Consultant at Alchemy Worx

Nice - at Alchemy Worx we find the meaning emoji convey depends entirely on the context used - semantic? illustrative? Makes a difference. Here is a handy guide with more tips for using emoji in your subject line http://bit.ly/1mOSVNS

22 days ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.