Poor A/B testing methodologies are costing online retailers up to $13bn a year in lost revenue according to new research from Customer Experience Management (CXM) platform, Qubit.

Whilst A/B testing – which lets retailers test the impact of changes to their websites – has often claimed to deliver sales uplifts, poorly executed tests often fail to deliver meaningful results and can even lead to a negative impact on sales. Properly executed tests, however, have been shown to deliver a potential 12% uplift in sales, totalling $13bn across the global revenue of US companies according to the Qubit research.

A/B testing involves comparing a change to a website against the original with consumers on a live website. The sales impact of the change can be measured by comparison to the performance of the original. However, whilst the concept is simple, there are a number of statistical tricks that can catch out the unwary, producing results that seem to show significant positive results where none are present or that understate positive impacts. Without a solid application of statistics to A/B testing approaches, retailers risk actually damaging their business through reliance on inaccurate testing results.

There are three key reasons for failures in A/B testing:

Insufficient statistical power – Statistical power is the probability that a test will accurately detect a trend. To ensure appropriate statistical power the sample size for a test must be carefully calibrated to ensure accuracy – a calculation that is often missed out in mainstream testing approaches.

Multiple testing – Running multiple A/B tests simultaneously – as advocated by many systems – significantly increases the likelihood of receiving a false positive result. Similarly, stopping tests as soon as they show a positive result significantly impacts on testing accuracy.

Regression to the mean – Over time, many apparently positive changes will see a declining return. This is due to the well-known but rarely applied, statistical phenomenon of regression to the mean – which essentially states that false positives are inevitable in small testing samples but that anomalous results will regress towards the average, accurate result over time.

Graham Cooke, CEO of Qubit, said: “A/B testing has been sold as a way to scientifically test the impact of changes to your website. However, as with all science, unless your experimental methodology is robust, the results of your testing will be meaningless. Many A/B testing approaches take a statistically naïve approach to methodology, leading to test results that are inaccurate at best and actively damaging at worst.

“Optimizing website performance is a top priority for advanced online retailers and testing is a vital part of making sure that optimization strategies are effective. However, if your testing is flawed then you might as well make changes at random in the hope that they’ll succeed”

Qubit Whitepaper: http://www.qubitproducts.com/content/most-winning-ab-test-results-are-illusory

Qubit infographic: http://www.qubitproducts.com/content/infographic-ab-testing-5-checkpoints-road-success


For more information:
MWW London
0208 875 7970

About Qubit:
Qubit is a big data customer experience management (CXM) platform. Its technology helps the twenty-first century marketer create personalised website experiences to drive enhanced conversion rates.

Qubit’s products collect huge amounts of information about users’ interactions with a website and analyse this to identify patterns of behaviour. They then use this understanding to deliver live personalisation, driving improved conversion rates and increased customer satisfaction.

The company’s suite of products brings together visitor analytics, tag management, AB testing and personalisation within a single integrated platform. This allows marketers to optimise and test websites in real time, without the need for specialist IT resource, helping to drive rapid and sustained ROI.

Founded in 2010 by four ex-Googlers, Qubit is headquartered in London with offices in New York. Qubit works with some of the world’s leading ecommerce businesses, including TopShop, Staples, John Lewis, Farfetch and Harveys Furniture. The company received $7.5 million funding from Balderton Capital in December 2012.

Qubit enables online marketers to:

1. Integrate with ease – Qubit consolidates many disparate technologies, plugging into a site via a single line of JavaScript and our Universal Variable data model; this one time integration creates a lifetime foundation for data collection and targeting

2. Test with confidence- Qubit combines quantitative and qualitative data to predict visitor behaviour and target high value user segments; this helps site owners prioritise how they optimise

3. Remove IT barriers – Qubit has a hands on interface that lets marketers and ecommerce professionals discover insights and deliver personalisations without hassling the IT team; this puts marketers in control and means they never have to miss a marketing opportunity

4.Deliver on-going ROI - Qubit is a low risk, cost effective and transparent solution that rigorously measures any change you make to a site, from A/B tests to live personalisations; this means site owners can trust their results and constantly justify investment

Published on: 1:04PM on 27th February 2014