This was made more difficult as the President was less available for fundraising events in 2012 and the average online donation was half what it had been four years earlier thanks to a combination of the poor economy and a drop in voter enthusiasm.

To overcome these problems the digital campaign had to be more innovative, so A/B testing of emails and other messages became far more important.

Winning with A/B testing

A majority of the online donations came as a result of email campaigns, which were constantly tested to find the most effective subject line and content.

For example, in one test on subject lines Showalter’s team found that the most effective iteration would raise $2.5m in donations, while the worst performing subject line would bring in less than $500,000.

Similarly, the team achieved a 5% uplift in conversions by A/B testing a long online donation form against a sequential format that asks for a little bit of information at a time, with the latter proving to be more effective.

This doesn’t sound like a lot, but you have to remember that over the course of the five months the donation page was live we raised a few hundred million dollars. So 5% of that is a huge amount.

Showalter also revealed that campaign tested different ways of encouraging voters to save their payment details online (similar to Amazon’s 1-Click method), so that future donations would be quicker. 

By changing the text at the end of the payment process from ‘Save your payment details now to make the process quicker next time’ to ‘Now save your payment information’ the conversion rate increased by around 20%-30%.

The reason for this is that voters felt that saving their details was just the next step of the process rather than an optional add-on.

Email tests

The Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total.

Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama.

Some of the tests achieved an additional $2.2m in revenue from sending the best draft compared to the worst draft. 

After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content.

Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history.

The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. 

This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue.

The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events.

And here are the seven lessons the team learned from A/B testing…

1. Don’t trust your gut

Showalter’s team found that conventional wisdom is often wrong, as are long-held best practices. It’s important to remember that you are not your audience, so everything needs to be tested. 

The digital team often tried to predict which would be the top performing emails in the A/B tests, but never got it right.

2. Ugly vs. pretty designs

To try and boost donations the digital team attempted to improve the design by making them look “prettier”.

That failed, so in response an “ugly” design was tested to see if that made any difference. This involved using yellow highlighting to draw attention to certain text within the email.

To the team’s surprise the ugly design actually proved to be quite effective, though the yellow highlighting had to be used sparingly as the novelty wore off after time.

3. Foster a culture of testing

Digital marketers need to check their ego at the door and use every opportunity to test something.

And when benchmarking results you should compare against your own previous campaigns, not always against the industry average. 

For example, consider whether you’re doing better than last month? And are you doing better than you would have otherwise without testing?

4. Keep a testing calendar

The Obama campaign had short and long-term calendars for national emails campaigns.

This included a ‘tests’ column to plan which experiments were attached to each email blast. Any blanks in this column were a reminder that they needed to plan in more tests. 

5. Circulate your test results internally

Showalter’s team circulated the results of all its tests via an internal email.

This helped to get buy-in from other departments and increase familiarity with the testing process. It also prompted good discussions and generated new ideas for tests.

6. Invest in your team

The digital team grew from a handful of people in spring 2011 to a department of more than 200, making it the largest single department within the Obama campaign.

Many staff members had never worked in politics before, but Showalter felt that experience is less important than aptitude and passion. Instead the staff learned about politics and programming on the job.

Diverse voices led to better content and analysis. When recruiting within digital analytics look for people with strong quantitative skills who are willing to learn.

7. Be human

A personal touch can be very effective in email marketing, For example, a subject line that simply said “Hey” consistently proved to be the most successful. In general shorter, less formal email lines were the most effective. 

Mild curse words were also successful, such as “Hell yes I like Obamacare” and “Let’s win the damn election.”