In case you’re a few years behind the times, you will be aware that Obama’s re-election campaign was a success.

But what is less well known is the detail of the testing process behind the email strategy that helped to raise more than $500m in online donations.

At Searchlove this morning Obama's director of digital analytics Amelia Showalter gave an insight into the A/B tests that optimised the campaign's fundraising emails and the lessons that the digital team learned as a result.

Showalter said that in a tightly fought election Obama’s campaign team knew they would have to top the $750m raised in 2008.

This was made more difficult as the President was less available for fundraising events in 2012 and the average online donation was half what it had been four years earlier thanks to a combination of the poor economy and a drop in voter enthusiasm.

To overcome these problems the digital campaign had to be more innovative, so A/B testing of emails and other messages became far more important.

Winning with A/B testing

A majority of the online donations came as a result of email campaigns, which were constantly tested to find the most effective subject line and content.

For example, in one test on subject lines Showalter’s team found that the most effective iteration would raise $2.5m in donations, while the worst performing subject line would bring in less than $500,000.

Similarly, the team achieved a 5% uplift in conversions by A/B testing a long online donation form against a sequential format that asks for a little bit of information at a time, with the latter proving to be more effective.

This doesn’t sound like a lot, but you have to remember that over the course of the five months the donation page was live we raised a few hundred million dollars. So 5% of that is a huge amount.

Showalter also revealed that campaign tested different ways of encouraging voters to save their payment details online (similar to Amazon’s 1-Click method), so that future donations would be quicker. 

By changing the text at the end of the payment process from ‘Save your payment details now to make the process quicker next time’ to ‘Now save your payment information’ the conversion rate increased by around 20%-30%.

The reason for this is that voters felt that saving their details was just the next step of the process rather than an optional add-on.

Email tests

The Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total.

Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama.

Some of the tests achieved an additional $2.2m in revenue from sending the best draft compared to the worst draft. 

After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content.

Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history.

The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. 

This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue.

The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events.

And here are the seven lessons the team learned from A/B testing...

1. Don’t trust your gut

Showalter’s team found that conventional wisdom is often wrong, as are long-held best practices. It’s important to remember that you are not your audience, so everything needs to be tested. 

The digital team often tried to predict which would be the top performing emails in the A/B tests, but never got it right.

2. Ugly vs. pretty designs

To try and boost donations the digital team attempted to improve the design by making them look “prettier”.

That failed, so in response an “ugly” design was tested to see if that made any difference. This involved using yellow highlighting to draw attention to certain text within the email.

To the team’s surprise the ugly design actually proved to be quite effective, though the yellow highlighting had to be used sparingly as the novelty wore off after time.

3. Foster a culture of testing

Digital marketers need to check their ego at the door and use every opportunity to test something.

And when benchmarking results you should compare against your own previous campaigns, not always against the industry average. 

For example, consider whether you’re doing better than last month? And are you doing better than you would have otherwise without testing?

4. Keep a testing calendar

The Obama campaign had short and long-term calendars for national emails campaigns.

This included a ‘tests’ column to plan which experiments were attached to each email blast. Any blanks in this column were a reminder that they needed to plan in more tests. 

5. Circulate your test results internally

Showalter’s team circulated the results of all its tests via an internal email.

This helped to get buy-in from other departments and increase familiarity with the testing process. It also prompted good discussions and generated new ideas for tests.

6. Invest in your team

The digital team grew from a handful of people in spring 2011 to a department of more than 200, making it the largest single department within the Obama campaign.

Many staff members had never worked in politics before, but Showalter felt that experience is less important than aptitude and passion. Instead the staff learned about politics and programming on the job.

Diverse voices led to better content and analysis. When recruiting within digital analytics look for people with strong quantitative skills who are willing to learn.

7. Be human

A personal touch can be very effective in email marketing, For example, a subject line that simply said “Hey” consistently proved to be the most successful. In general shorter, less formal email lines were the most effective. 

Mild curse words were also successful, such as “Hell yes I like Obamacare” and “Let’s win the damn election.”

David Moth

Published 28 October, 2013 by David Moth

David Moth is Editor and Head of Social at Econsultancy. You can follow him on Twitter or connect via LinkedIn

1719 more posts from this author

You might be interested in

Comments (3)


Zach @ ReferralCandy, Tea Drinker at ReferrlaCandy

Really loved the counterintuitive insights from the A/B testing of the emails:

1. Mild curse words performed better
2. Informal subject lines (starting with Hey.. ) performed better
3. 'Ugly' designs worked better
4. Yellow highlighting worked better (if used lightly)

It's a good reminder for marketers to adopt a testing mindset instead of going with your gut.

Thanks for the post! I would love it even more if Obama's digital team released the full results of the A/B testing. I'm sure that they would have tonnes of insight from the millions of emails tested.

almost 5 years ago

Tim Watson

Tim Watson, Email Marketing Consultant at Zettasphere

Showalter is quoted saying that of the $500 million from email roughly $200million of that was due to the email test. Which is a huge email test success story.

The other two key factors were list size and frequency.

I got to speak to Toby Fallsgraff who was part of the Obama digital team and he shared with me their test strategy and how they did Frankenstein tests.

What he told me is documented here

almost 5 years ago


Paul Hilton

Looks like someones been reading "A/B Testing - the most powerful way to turn clicks into customers"

Interesting article which as a company we're only just adopting. It's amazing how a simple test can quash a 'design by committee' page.

almost 5 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.