For many, Multivariate Testing is an as yet untried website optimisation tool, but it isn’t as complicated as it looks!

Here are three considerations for choosing an MVT tool.

Multivariate Testing (MVT) is a great way of accelerating your user experience programme with quick-to-market changes and in-depth analysis to back up site improvements.

Through the years the key question I hear most, and perhaps more difficult to answer than justifying the need to do MVT, is deciding on which tool is right for the business and how to set the business up for success.

MVT is your racecarThere isn’t a simple answer and certainly not one that could fit into a blog post so here is a brief guide from my experience around three things to consider before embarking on supplier selection.

1. Internal capability

Weighing up resource against budget, this is the most important factor in deciding the tool and operating model. 

The old 80/20 rule certainly comes into play for MVT; the majority of your budget should be focussed on resource.

Your team will decide what to test, based on where the need is for your website. Your team will create the test plan, they’ll decide on the inputs and measurements. Your team will interpret the results and create additional improvements. The tool simply makes this happen.

Think of MVT as your race car; the more experienced the driver and the better the support team, the greater chance of success, test after test.

Typical roles around MVT to consider are:

  • MVT Lead – this is your race car driver, the person that gets it and makes it happen. They’ll manage the brief, test setup, launch, monitoring and create actions from insights. A must have to lead your MVT to success.
  • Analyst – as with any data, you can cut the results any way you wish; however interpreting the results is the most important part of MVT. A must have in your MVT team to learn from and improve your tests.
  • Developer – depending on the complexity of the test, you may need the tests coded up. However the importance of this role depends on the type of tool and solution (and whether you get this level of access).
  • Designer – they’ll take the lead in turning ideas into reality. Well worth ensuring designs are coherent with your existing site and meet expectations to really make MVT work for you.

The number of people in each role depends on the size of your business and ambitions. If you can identify one or more of these roles through internal resource this could change the dynamic of the tools you consider.

If testing button placements and copy is the extent of what you want to do (for example your website is fairly mature and/or design and layout is heavily guarded by your branding team) then a heavyweight MVT tool and expansive MVT team is overkill.

You’ll certainly get out of MVT what you put in and getting the right people into your MVT team will pay dividends, no matter whether that’s through a third party or internally.

2. Budget

There are a wide range of tools available, from fully managed testing tools and solutions, to self-managed entry level tools. You could even develop something yourself. Whilst available resource will help define the required budget, so will the level of hands-on responsibility that you wish to take on.

You could opt for a fully managed and high cost solution, which has the benefit of not tying you down internally and allowing your or someone in your team to oversee and provide direction.

You could go for the free model and bulk up your internal resource to drive and support this. Or perhaps an in-between mix of internal resource and third party support.

Cost and lack of belief in the benefits are common deterrents for spending on MVT. But used well it can provide some astounding results. Triple figure ROIs, triple digit percentage sales uplifts.

Done in the right way, MVT can reap in a lot of the cash your site actively leaves on the table, reduce wasted marketing spend and keep customers loyal in a time of e-commerce proliferation and abundant choice of where to shop.

Don’t expect miracles from early tests but get it right and you should see payback well within a year.

If your website is non-transaction don’t let this deter you. Increasing engagement, loyalty and sign-ups are all common goals for non-transactional sites. If you can attach a value to each key action then you too can create an ROI from MVT.

3. Intent

How far are you down the line of website optimisation? If you’re just starting out, then MVT perhaps shouldn’t be the highest item on your list.

The 2010 Econsultancy Conversion Report is a useful resource I’d urge you to read before embarking on conversion optimisation. Analyse, test, optimise and measure are all key tenets of conversion optimisation. MVT certainly falls into the test and optimise portion of this process.

So if you’ve covered the basics around analysing problem areas around your site then MVT can help resolve issues and improve conversion rates leading to better retention and loyalty. Indeed there are other tools available aside from MVT to help optimise conversion which you should consider however these are out of the scope of this post.

The level of testing will be derived by many factors including business and website maturity, budget and expectations; these all play a part in the right solution so it’s worth considering your intent for MVT as part of a wider programme.

I’ve not come across a silver bullet for improving conversion however MVT provides the framework to deliver improvements. Get the framework right and the rest is down to the direction you take it in. Free tools can deliver as much ROI as fully serviced tools, given the right structure.


There’s a (free!) resource well worth visiting for comparing MVT tools to give an idea of what’s available, types of models offered and capabilities.

You can then use this brief guide to help think about what will be right for your business and start taking the first steps.

For a more in-depth view and additional insights, the MVT buyer’s guide is a fantastic walkthrough of the broader and more detailed aspects of MVT. Choosing the right tool is certainly no small feat.

What model have you chosen for your business? Or, if you’re considering MVT, does this change your thinking?