We're seeing growing interest from site owners in doing A/B split testing and multivariate / multivariable testing - for landing pages, for product pages, for buying processes, for e-mails and so on.
But who are the service providers out there specialising in offering such solutions?
We (E-consultancy) are looking at doing some specific content and events (roundtables, training and so on) around the topic of testing and optimisation as we see it as an enormously important topic going forwards.
At this stage we’d be interested to know whether you think the topic is of interest, and also who you know out there that specializes in this area? Have you got any experiences, recommendations or suggestions to share? Feel free to put yourself forwards if you are a supplier / agency!
Following is a list of companies currently on our radar:
Specific multivariate / multivariable / split testing tools and services:
- Adlucent - http://www.adlucent.com/optimize/landing-page.html
- Google Website Optimizer - http://services.google.com/websiteoptimizer/
- Memetrics - http://www.memetrics.com/
- Offermatica - http://www.offermatica.com
- Optimost - http://www.optimost.com
- SiteSpect - http://www.sitespect.com
- SplitAnalyzer - http://www.splitanalyzer.com
- TaguchiNow - http://www.taguchinow.com
- Vertster - http://www.vertster.com/
Site conversion optimisation solutions (typically continuous learning approach):
- [x+1] - http://www.xplusone.com/
- Kefta - http://www.kefta.com/
- Maxymiser - http://www.maxymiser.com
- Touch Clarity (Omniture) - http://www.omniture.com/touchclarity
- ATG Campaign Optimizer - http://www.atg.com/en/products/campaign_optimizer.jhtml
- Wunderloop - http://wunderloop.com/en/?page_id=13
(NB I’ve left out a whole load of online advertising optimisation solutions, including the behavioural targeting crowd here – in theory these can of course be used to serve and optimize content on your own site, not just your ads on other sites.
I’ve also left out a number of optimisation solutions that are largely focused around search engine marketing e.g. landing page optimisation analytics built into a bid management tool. The above multivariate testing tools will often be used for search marketing optimisation of course.)
Any thoughts, suggestions?
Ashley Friedlein
CEO
E-consultancy.com
Ashley Friedlein is CEO and Co-founder of Econsultancy. Follow him on Twitter (3,800+ followers) or connect via LinkedIn (4,500+ connections) or Google+.
Head of Usability & Conversion at PRWD
9:42AM on 16th May 2007
I would definitly welcome training and events geared towards this form of testing and optimisation. I have previously come across a useful <a href="http://www.marketingexperiments.com/improving-website-conversion/multivariable-testing.html">article on multi-variable testing</a> which provides comprehensive information and links to both providers and other articles on this subject.
It appears you have already identified most of the software and services provided, but it may provide some more useful articles and findings which you can use when you start pulling together your training/roundtables etc.
In my experience at Littlewoods Shop Direct, for the few times when it was feasible to carry out A/B testing it proved very worthwhile and allowed the business to make key commercial decisions as to what provides the most optimal user experience.
Finally I totally agree that multivariate and A/B testing will be a key method of testing and optimisation in the future, along with other usability testing techniques such as eye tracking and user testing.
12:00PM on 17th May 2007
Thanks for the coverage and also Paul, glad to see you feel this will be useful in the future, we are seeing huge growth in interest currently as more and more people come towards fully exploiting other traffic driving and retention methods.
Just a quick note to add that Maxmyiser also offers Multivariate, split and A/B testing tools separately to our optimisation consulting, we can service all types and sizes of clients.
"I work for this company"
3:34PM on 17th May 2007
Please add Amadesa to the mix. We are a business optimization solution with focus on the ecommerce, lead generation and subscription-based markets. Our combination of behavioral targeting, a/b and multivariate testing, analytics and user interface design alterations is unique in the industry, and enables us to support a wide variety of client-specific needs (product & price testing, landing page optimizations, checkout/billing path improvements to name a few).
It's great to see movement forward on round table type discussions. Are you considering webinars, events, tradeshows, etc? I'd be more than happy to help where I can.
Pete Olson
VP Marketing
Amadesa, Inc.
Technical Director at Box UK
4:18PM on 17th May 2007
We don't specialise in A/B Testing, but can't resist a quick plug (our tool offers quite "accessible" A/B Testing from £2.50/month, so can't complain!).
We've also recenly 'eaten our own dog food' and run a series of A/B Tests on the clickdensity (http://www.clickdensity.com) site, which I'll be writing a case study on soon.
I'd personally be really interested in any content/events that you create on this topic.
5:53PM on 18th May 2007
We at SiteSpect embrace formalized training, and believe that discussion in the area of testing and optimization is a wonderful idea. Along those lines, an important consideration is inclusion of data from multiple sources – not just web sites, but email campaigns, PPC/banner campaigns, fulfillment systems, etc. Only through this vantage point can Web marketers ensure optimal performance across each segment, and at each point in the customer lifecycle. Seamlessly integrating these data sources is key to understanding the overall impact of multivariate tests, and ultimately to maximizing the ROI of optimization efforts. We would be happy to participate in training and discussion on these topics.
5:18AM on 21st May 2007
Memetrics has been focused on applying best practice multi-variate testing and optimization methods for over 8 years now. It is great to see the market embrace an approach that we have seen deliver consistent value.
It is worth considering that the topic is actually much broader than just site side optimization. Inherently, all digital media (and digitally enabled customer processes) can benefit from the same testing and optimization process applied to sites. We have run successful programs across site, e-mail, direct mail, telemarketing, SEM, on-line media etc.
It is also worth noting that while technology is a critical factor for success, equally important are the marketing methods used to drive the capability.
Digital media provides rich opportunities to improve marketing effectiveness and gain actionable customer insights.
We'd be delighted to share experiences and lessons learned along our optimization journey.
Hikaru Phillips
8:02PM on 8th August 2007
Hi Ashley,
Thanks for the question. How about a buyer's guide on multivariate testing providers? There's a real dearth of information when it comes to vendor selection.
thanks!
Andrew
CEO at Econsultancy
9:42AM on 9th August 2007
Hi Andrew
Yes, I think that's definitely something we should do - we might broaden it slightly to 'optimisation technologies/solutions' but certainly something is needed.
Ashley
6:20PM on 23rd August 2007
Please include Sokel Inc. which has released Sokel Choicepoints - a cost-effective solution that enables marketers to run simple A/B or more complex multi-variate tests to understand which of many different combinations of creatives performs best. Sokel Choicepoints also supports learning where creatives are served out in the proportion they convert.
Thanks
Sridhar
1:11PM on 7th September 2007
Hi Ashley,
The article was quite informative. I liked it !!
Could you tell us how Fractional Factorial testing is actually implemented ? What are the calculations involved ?
If you could point me to some links, that would do too.
Thanks in advance. :-)
Cheers,
Ashlak.
CEO at Econsultancy
4:24PM on 7th September 2007
Hi Ashlak (BTW is that Ashley in another language? ;) )
In answer to your question "Could you tell us how Fractional Factorial testing is actually implemented"...
No. No idea. I'm not even sure I understand what Fractional Factorial testing is, let alone how it works.
Sorry!
Ashley
6:53AM on 11th September 2007
Hello Ashley,
Not a problem... Thanks for the reply, anyway !! :-)
Fractional Factorial testing is the process of testing a "Fraction" of all the "factors" ( "factors" are the elements on a page such as Headline, Image, Promo Text, etc... Could be just about anything ) on page to test what's working and what's NOT !!
Its Primary purpose is to Improve Conversion Rates !!
If anyone else has something to ADD or CORRECT, the floor is yours !! :-)
Cheers,
Ashlak
PS - Ashlak is not a name in any language !! I like it for some reason !! :-)
12:56PM on 2nd October 2007
I'd defintitly be interested if you did a round table or show case event, as I think there are a lot of people in a similar position - either just starting to get into it or thinking about it. But I'd also be interested in a session on how it impacts your design process. If anyone could paint a solid picture of what is required to run a wesite based on a continual optimisation process I'd definitely be interested in that too. I guess the vendors are part of it, but there's a fair bit else that I'd be interested in as well...
Director at WhyCommunicate?
10:02AM on 4th October 2007
Hi Ashley
This sounds like a great idea. Certainly we are finding our clients are now showing an interest in this type of activity particularly when we have been teaming this with the development of landing pages as part of a wider PPC intiative.
As a result we have seen some very significant improvements in CTR from landing pages.
We would welcome more exposure in the market of the benefits of this tyep of solution.
Garry
3:39PM on 26th November 2007
Hi Ashley,
Last December we launched an article on multivariate testing that featured on the Alexa.com homepage as a Top 3 'Mover & Shaker', as well as hitting the home pages of Digg & Delicious - interest in conversion optimization is huge!
Our recent presentation at A4UExpo in London was also very well received - so go for it!
Warmest Wishes,
Ben
P.S. Shoot me an email if you want any help, feedback or advice with this.
5:19AM on 24th September 2008
I use Google Adwords to test print ad headlines. There's a 25 character limit, but that usually enough for your standard headline.
The thoery is that a headline that works well online will probably work in offline media. So far it's proven to be true.
12:12PM on 9th October 2008
I just read a book called Supercrunchers. They rave about a company called Offermatica. Every heard of them?
4:42AM on 30th January 2009
Hey you forgot about these guys: http://mixpanel.com
3:31PM on 14th March 2009
I have found Google's website optimizer tool perfect for the job. Only drawback is possible lack of advanced reporting facilities - but considering it's free, I'd rather use that than spend £1500 minimum a month with MV companies.
12:28PM on 27th March 2009
Hi,
Watch this space - I'm doing some research into the whole usage, drivers and inhibitors for this type of technology and hope to work with e-consultancy to publish some of it when completed.
I find it incredible that more people aren't taking advantage of this technology, especially in a downturn, as it can improve value from any traffic significantly.
Craig.
Director at NetInsight eBusiness Consultancy
4:57PM on 5th May 2009
We have seen this industry sector grow considerably over the last year and it seems more and more of our clients are lookingt o us to provide this service.
We have used Google Web Optimiser extensivly and can't find a great deal of fault with it. However we are always looking to try other systems, but can't justify the huge cost to the client as yet.
Any suggestions would be great
Cheers
Dan
3:24PM on 19th May 2009
Hi Dan,
I agree Google Optimizer is good as far as a free tool goes, but it is not without its faults. I have recently written a review of this tool in my blog, which you can access by clicking on the link in my name.
Best regards - Jeremy
Director at NetInsight eBusiness Consultancy
3:34PM on 19th May 2009
Jeremy,
Thanks for you feedback, i'm working my way through your blog now and taking in all your points. I'll get back to you with my thoughts!
Thanks again
Dan
6:03PM on 16th June 2009
If you ever update your list, please add Widemile! We've been around for many years but are just now starting to push our testing and optimization tool to the masses.
We've been doing full service work mostly, but now are offering the same tool we've been using internally to enterprise customers.
I've been writing about testing and optimization techniques on my blog too. I'd love to get some comments from people like you who have experience with optimization.
2:19PM on 25th August 2009
Same as Billy - if you are going to update your product list I would like to suggest adding the new multivariate testing tool from Oban Multilingual - GlobalMaxer.
Since business practices and markets vary from culture to culture, our product has been specifically designed for testing and optimising websites within their international contexts.
CEO at Conversion Rate Experts
2:51PM on 1st March 2010
We've just published a comparison website of all the Multivariate testing tools. It includes user reviews too... http://www.whichmvt.com You're comments and feedback are very welcome.