{{ searchResult.published_at | date:'d MMMM yyyy' }}

Loading ...
Loading ...

Enter a search term such as “mobile analytics” or browse our content using the filters above.

No_results

That’s not only a poor Scrabble score but we also couldn’t find any results matching “”.
Check your spelling or try broadening your search.

Logo_distressed

Sorry about this, there is a problem with our search at the moment.
Please try again later.

In today’s hyper-competitive online market, success is all about understanding and using your data. However, statistics is a complex subject matter and most people working in marketing, even online, don’t come from a statistics or mathematical background.  

Despite this, the majority of 21st Century Marketers are coming to appreciate that there is a need for a robust, data-driven approach to maximising returns on marketing investment.

As the economic landscape has worsened for retail, every minute spent and every pound invested has to work hard and prove ROI in order to justify its activity. Declining margins and increasing traffic costs mean that the performance improvements inherent in the clever use of data are now more valuable than ever. 

Conversion Rate Optimisation is now common place across most retailers, with 27.5% of ecommerce sites in the UK now undertaking an AB testing programme in some form to drive incremental revenue.

While this means that three-quarters of online retailers are still not effectively measuring the impact of their marketing spend, it means that more than a quarter have accepted that data is key to online success.

However, it’s important to remember that not all AB tests are equal.  Any form of testing is only as good as the methodology and the data that underpins it.  

This might seem obvious, and is easy to say, but what are the keys to a truly successful testing approach?

Start with a data driven hypothesis

One of the most common causes of testing failure is starting with a poorly formulated hypothesis.  

Testing a hypothesis derived from a close study of valid data is more than seven times more effective than testing a hypothesis based on a hunch.

Your involvement with data really needs to start before you open your testing toolkit – you need to look at information about what your users are doing and try to establish what the roadblocks are to conversion.

Only once you have a hypothesis that seems to be supported by the data should you start testing solutions.

Measure overall conversions, don’t look at micro-goals

AB tests should be focussed on driving incremental conversions, not on meaningless metrics that might not contribute to revenues. Click-throughs and other vanity metrics are not a meaningful measure and should never be the basis of a testing approach. 

For example, craft site Etsy implemented infinite scroll on its site in an effort to increase the average time on site. However, the result was a negative impact on other more valuable metrics such as product pageviews – by testing for a meaningless metric the result was a downturn in revenue generating activity.

Don’t turn your test off too early

The volume of testing data is critical when it comes to meaningful results. Randomness in data can seriously compromise small sample tests. If you flip a coin 10 times you won’t necessarily get five heads and five tails each time.

The rough guide is that, in order to generate sufficient ‘power’ in your testing (i.e. enough data for the test to be meaningful), around 5,000-8,000 events of a success metric need to be recorded in both the variant and control of a simple test.  

Within this, you need to look at the number of variables that you’re testing. The more variants you have the longer it will take you to reach a meaningful sample size (5000* the number of variations). Having multiple variations also affects the analysis of the data.

If you flip a coin ten times and repeat that 100 times, you're going to get some "unlikely" sets of coin flips.  

AB test your MV test

If you are running a Multivariate test you should look to AB test your results to ensure accuracy. If you test a large number of variants simultaneously then the likelihood is that one of them will appear to succeed by random chance.  

You should therefore always AB test the winning formula from an MV test just to ensure that the results are replicable before declaring it a winner.

Define your segments in advance 

As well as basing the hypothesis of your test on data-backed insight, you should also define the audience segment that you want to test for upfront. Some testing approaches advocate running a test and seeing which segment wins but, because of a problem known as overfitting, this can lead to inaccurate and misleading results.

All of this points to a set of basic rules for testing.  If you work in ecommerce and have recently run a test which had less than 1,000 conversion events, had more than four variants or was not driven against a revenue related metric, then your best bet is to start again and re-run your AB test using the advice above.

Ian McCaig

Published 17 July, 2013 by Ian McCaig

Ian McCaig is Founder at Qubit and a contributor to Econsultancy.

29 more posts from this author

Comments (1)

Avatar-blank-50x50

Zac

Great article. I just began to use A/B Testing in my SEO research. It has been working in determining what I should be focusing on. The other tips provided are also useful, and it's important for marketers to test their plans before initiation. Can lead to improvements and risk reduction.

over 3 years ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.