Multivariate Testing Buyer's GuideTo coincide with the recent release of Econsultancy’s Multivariate Testing Buyer’s Guide, we’ve been talking to some companies about best practice and useful advice for those thinking about investing in this area and looking for an MVT vendor. Below, we’ve included tips from client-side practitioners at Belron, the FT, Lovehoney, Telegraph and Tesco. 

Whether you are just beginning to realise the importance of investing in good testing technology or you are already carrying out a comprehensive conversion rate optimisation programme, these tips will help you get the most out of your tests and stay on top of your game.

What do you see as the greatest benefits of multivariate testing?

Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron

With MVT, no idea is a bad idea. Remember that really annoying person who always brings dumb ideas to your meetings or won’t compromise? Let’s call this person Dick. If he has one of his great ideas, include it in your test plans and show the results. It’s hard to argue your idea is the best one, if it takes less money. Dick will also stop making stupid suggestions when he realises he isn’t having much luck in the tests.

If you have problems with three hour meetings discussing button colours or arguing with time wasting morons about how to design a page, you should do MVT. This has removed endless useless and life sapping meetings from my calendar. We no longer have to agree one design, only for the senior exec to change it yet again. Every idea is considered, added to the test and measured. The results speak far more clearly than any opinion or ego within my company.

What advice would you give to companies looking for an MVT vendor?

Craig Sullivan

When the salmon return from the sea to the river, they stop feeding until they spawn. If they’re not interested in feeding, how do people catch them? It may take all day, with a salmon sitting behind a rock, to find the right lure and the perfect place to drop it. You might catch a fish the first time or spend all day with nothing to take home. You’ll try endless combinations of lure, bait and placement to get any reaction and just when you are ready to give up – bang – a flash of line tells you that you’ve finally got some attention.

MVT is like having a thousand lines in the water, to find the lure that will catch the most fish. You’re welcome to patiently try everything in your repertoire but those that experiment, they get their fish quicker and more easily than everyone else. If you have less budget, a competitive market or both – make sure you have the edge when you’re out fishing.

Sandra White, Head of Optimization, The Financial Times

Think about what you want from MVT. In our case, MVT doesn’t really cover what we do. It is more about having a testing platform that allows us to optimise our site, which is an ongoing process. Having control over whole pages is therefore preferable to being limited to ‘test areas’.

I would also recommend looking at how much account management is included, and whether this covers help with testing strategy or any consultancy. Also, what are the workflow limits in terms of getting tests up? Can simple tests be set up by anyone in-house with some basic technical knowledge? Bear in mind that testing also requires design resource, which most vendors don’t supply.

Matthew Curry, Head of Ecommerce, Lovehoney

If you’re able, conduct tests that optimise on a monetary figure, such as revenue or even better, profit. That way you can make an easier judgement on how much the tool is worth to you. MVT isn’t a magic bullet, it will require a lot of effort to get results. Don’t budget on how much extra revenue you could theoretically make, remember to be pragmatic.

Depesh Mandalia, Senior Marketing Manager – Personalisation & MVT, Tesco

This depends on the level of expertise within the business and the level of access to your backend code. Some vendors offer a managed service, taking away the complexities involved in setting up tests and many also minimise the level of change required in your backend code for each MVT you run. These can have a huge impact on the decision points. I would also stress that it is worth looking at the right vendor for where your business is and if choosing a more basic solution, you always have the opportunity to move up to one of the more advanced (and usually more costly) platforms once you’ve operationalised MVT within your business.

Emilija Vilkyte, Head of Digital Revenue Optimization, The Telegraph

Before choosing an MVT vendor you need to be very clear about the type of tests you are looking to run, how often, how much internal resource you have got and how quickly you expect to see the results, as there are a lot of different products and propositions in the market. Also be clear where the tests will run (same domain vs. subdomains and third parties).

From your experience, what kind of approach has been successful and what has been unsuccessful?

Depesh Mandalia

Start simple and don’t expect business changing uplifts early on. The beauty of MVT is the last letter of the acronym – Testing. MVT allows you to test many ideas within a relatively short space of time, speeding up the learning process and development time. From experience, MVT never ends. In addition, you won’t always create improvements – learn and improve. There is always something to improve, something to tweak, new ideas to try out and with any business, as the business dynamics change, as do the opportunities MVT can leverage. The key for any business is to involve the right stakeholders. MVT can make those less involved feel like they’re losing control. Instead highlight the reduction in guesswork, in wasted development time trialling ideas and money saved in sifting out the bad ideas from the good as quickly as possible. Keep the key stakeholders engaged and involved and the whole process becomes easier.

Emilija Vilkyte

As long as you don’t change your mind and test spec in the middle of the process everything will be fine. Document all possible user journeys and scenarios – this will help you understand main problem areas. Be very clear ‘where you are’ and ‘where you want to be’ when providing instructions to your MVT partner. Create a test roadmap to manage internal expectations. Make sure you test variants before test goes live on all browsers and platforms.

Craig Sullivan

Don’t try to test too many variables, especially if your traffic isn’t huge to your site. For example, does testing a different shade of blue on your button really drive behavioural change? You’re much more likely to understand what resonates with customers if you are bolder and try new things. Avoid testing subtleties or small changes that will eat into your testing time. There isn’t much point in taking eight months to test your homepage, as during that time you’ve not had the final uplift. Like Agile, do your testing in smaller batches and learn from the results to drive your next test. Small, fast and likely to sway outcomes is best.

Sandra White

Testing and optimisation for the FT is about taking the guesswork out of everything to do with our site, whether it’s to do with conversion, engagement, usability, personalisation. Eighty per cent of the work of testing is about identifying worthwhile opportunities, otherwise you end up testing for the sake of it. It isn’t about changing button colours – the important thing is keeping business objectives in mind when planning testing.

Matthew Curry

There’s a lot of MVT press releases about how companies exponentially increased their revenue through a single small change. This doesn’t often happen, so don’t expect to recreate those successes, especially if you’ve already followed a lot of best practice. Instead, look to transformational testing: your message, your layout, photography, checkout. Change big things first, then when you think you’ve got those all working, you can concentrate on smaller changes.

Are there any other tips you would give to companies who are thinking about investing in MVT technology and related internal infrastructure?

Craig Sullivan

Don’t forget segmentation. If you can’t use your testing tool to segment, you should set up your web analytics to track this. Although you can find one design that converts best for everyone, this isn’t the optimal way to get a lift – look at which creatives most appeal to different audience segments. More companies now offer you the ability to then serve these to visitors – it means you are always showing the best converting creative to visitors, based on historical data.

Depesh Mandalia

Finding the right MVT tool for your business is something only you can define, by thinking about your current resource, current understanding, the time you are prepared to commit to MVT and where MVT sits in your order of priority against other web improvement projects.

Sandra White

The technology is only as smart as the information it can be given. To use it to its full potential, it needs to be integrated with other systems and across site, so that this can be fed into the decision-making layer. This can take some leg work. But once you start a proper programme of testing and optimisation, though, the potential is almost endless, assuming you have decent traffic volumes.

Emilija Vilkyte

Negotiate on the price as much as possible – there are ways to save money by limiting the number of tests a month, doing some data or design work yourself, removing some feature from your account e.g. there is no point to go straight to personalization if your site’s usability is fundamentally wrong.