Companies are collecting more data than ever about how their users interact with their websites, and thanks to sophisticated yet easy-to-use tools, techniques like A/B testing are accessible to even the smallest of businesses.

But when it comes to creating great user experiences, are companies being blinded by data?

The value of analytics data is without question, and countless companies use quantitative analysis of this data to improve their user experiences.

The numbers don't lie: changes informed by quantitative analysis are used all the time to reverse negative trends like shopping cart abandonment and produce higher conversions.

Unfortunately, the efficacy of quantitative analysis leads far too many companies to underestimate the value of qualitative analysis. Usability testing with focus groups, for instance, can be time-consuming and expensive, and if not run properly, produce misleading results. But that doesn't mean focus groups are not worthwhile.

One of the biggest problems companies face when they rely too heavily on quantitative analysis is that they assume they understand why changes they make in response to it produce the results they produce.

While it's often possible to create reasonable hypotheses, without talking to users, the answer to "Why?" is often never definitively known. This is particularly true with A/B testing, for instance, where small, seemingly minor changes produce not insignificant increases or decreases in important metrics.

Some argue that understanding why these changes produce the results they do is unimportant. If conversions are where a company believes they should be, for example, understand why something works is not as important as the fact that it works.

But as more and more companies seek to create good user experiences across channels, and user experience becomes an inseparable part of overall customer experience, companies would be wise to recognize that for all of its value, data and quantitative analyses can still be ambiguous, inconclusive and misleading.

The qualitative information that comes from talking to real users and customers can provide an invaluable layer of context and understanding that companies need to create the best experiences and identify ways to stay ahead of their competitors.

Patricio Robles

Published 1 June, 2015 by Patricio Robles

Patricio Robles is a tech reporter at Econsultancy. Follow him on Twitter.

2642 more posts from this author

You might be interested in

Comments (2)

Chris Rourke

Chris Rourke, Managing Director at User VisionSmall Business Multi-user

Thanks Patricio for a nice summary, I agree that quant research such as A/B or MVT (if you have the traffic volume) can provide strong compelling evidence IF the studies have been set up and interpreted well (thats a big if BTW). However you can only speculate as to the WHY and, more importantly, even if you determine that B is better than A according to the data (and are happy to not know why that is), you have no insight into what C, D or E can look like . It relies on variations offered to test and if the 'ideal' solution is not one of those, you'll be none the wiser. At least with people involved you get clues as to the ideal and very often it is not just what is there on the screen, but what is Not there. The qual research such as usability testing (rather than focus groups) will help to find the errors of Omission (what is missing) rather than just the errors of Commission (whats right or wrong on the page). So, like in many aspects of life, a balance is needed. In this case between well applied quant for the more deconstructive analytical means and qual for the exploratory approach.

about 3 years ago


Darren Ward, Director of Product Marketing at User Replay Ltd

Combining a strong platform of data with the ability to do qualitative analysis is really key to improving customer experience. Solutions such as UserReplay enable this by using a powerful dataset to direct you to the most relevant qualitative experiences to review so you can answer the "why" questions. It is then important to use the data to quantify these experiences based to understand the business impact of CX issues.

about 3 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.