It's one of the most powerful tools in an online publisher's arsenal but it's one that few make effective use of. I'm talking about multivariate testing.
As part of its effort to turn its business around, Yahoo is using multivariate testing extensively on its homepage to figure out which combinations of content and features produce the best results.
According to Yahoo's new CEO, Carol Bartz, Yahoo has tested 141 different versions of its homepage. As one of the most trafficked homepages on the web, Yahoo is hoping that it can benefit by figuring out which of the homepages it's testing keep users around longer, not only boosting the value of the homepage itself but potentially giving Yahoo a greater opportunity to get users engaging with all of its services.
What's truly amazing: according to PaidContent.org, Yahoo's homepage currently uses 33 different code bases (that's something Bartz wants to change). That means that its testing may be quite complex depending on how different the homepages being tested are from one another.
Needless to say, that Yahoo is using multivariate testing on an overly complex homepage that is built using a less-than-ideal hodgepodge of code bases hints at just how important Yahoo feels getting its homepage 'right' is.
And that's a good lesson for those of us whose homepages are a bit simpler. Thanks to the wide availability of multivariate testing tools, some of which are free, there's no excuse not to get into the act. Increased user satisfaction, usage growth and increased revenue are just some of the benefits I've personally seen achieved with even the simplest of A/B testing and it will be interesting to see what the results of Yahoo's multivariate testing are on the Yahoo homepage, and its bottom line.
Photo credit: Yodel Anecdotal via Flickr.
Patricio Robles is a tech reporter at Econsultancy. Follow him on Twitter.
Recommended reports
Online Measurement and Strategy Report 2011
An analysis of the online measurement industry based on a survey of digital marketers. The 67-page report and 29-slide summary presentation contain valuable insights into the use of paid-for and free web analytics tools. Beyond web analytics, the report also looks at a range of data sources and information requirements, and the extent to which companies are able to develop a coherent measurement strategy.
3:38PM on 26th March 2009
Multivariate testing is pretty amazing :) Good to see them doing it.
Would be interesting to know which company's providing them with the test tools. I'd personally recommend Google Website Optimizer as it's easy and free (wrote about it here http://lenalindstrom.blogspot.com/2009/03/how-to-increase-conversion-with-small.html). Drawbacks being no advanced reporting tools and being able to run multiple and really quite sophisticated big tests.
9:28PM on 26th March 2009
The technology was built internally. We looked at commercial services but none of them would work for our purposes.
(I led the team that developed the experimentation system.)
2:30PM on 30th March 2009
It'll be interesting to see if they do any segmentation within the MVT. That is, understanding that certain MVT combinations work better with some segements of their audience than others.
Agree that MVT is underutilized and underappreciated. It ranks up there with organic SEO and email marketing in terms of really good ROI.
GWO is a great tool which we use. Starting to look at others to see if we can do MVT within segments (demographics, geographies, etc.) to further improve conversion rates.
11:22AM on 31st March 2009
Actually, yahoo! have been doing multivariate testing for years using some pretty nifty in-house software, and it works well for them because with their volume of traffic, they get results quick.
But that's no reason for everybody else not to do it as well: it's now easy and free, it just requires some planning (but we all plan new feature and design releases propeorly, right?) ;-)
Advisor at The Lost Agency
5:13PM on 15th April 2009
Im impressed, google uses mvt for its search results, so the website with the worst usability MSN.com obviously should take notice and start reducing some of those damn popup ads, flash ads, and animated mouse overs...
11:47PM on 22nd June 2009
Hi Bob
im fascianted to hear that Yahoo! developed their own in house software for multivariate testing. Would none of the off-the-shelf systems works for Yahoo?
Do you have any plans to release the software in much the same way as Google has done with the Website Optimiser?
thanks
Russell