There is an ever-urgent need for companies to both acquire and retain online users, especially in today’s economic environment. Our new Usability and User Experience Trends Briefing highlights this, but which is best for achieving reliable results - internal or external testing?
Econsultancy estimates that client-side companies spent around £214m on usability and accessibility during 2008, which demonstrated a massive year-on-year growth rate of around 20%. This is due to an ever-increasing realisation that a commitment to the user experience is a key element to online success.
A recent trend that we've noticed is the development of internal testing departments within client companies, rather than outsourcing to agencies. It sounds like a forward-thinking move, but should we be questioning the benefits of this approach?
In-house testing is less expensive than outsourcing in the long run, assuming you do enough of it, and it helps any company to understand how usability fits into their business model. But, although data can generally be collected quickly, there are many barriers to success, such as a lack of industry knowledge (which can make a thorough assessment or understanding of results extremely difficult). Equally, other restrictions such as time, internal issues and conflicts, and a lack of specific technology can hinder a company’s efforts.
By contrast, outsourcing user testing to specialist agencies can deliver the benefit of results being generated in an unbiased environment, with industry specialists who usually have far greater experience and knowledge (along with better resources and the latest testing equipment and related technology). External testing generally has no restrictions placed upon it by the client and as such, results can be gathered quickly and presented accurately.
While internal testing can save money, it sometimes inadvertently cuts corners, meaning that results can easily be affected. Inaccurate data, upon which important decisions are based, which may ultimately damage any overall online performance.
We recognise the value of outsourcing in order to maximise testing, and know that some of the more savvy companies employ usability agencies on a retainer basis to guarantee ongoing testing of their websites. But there's still some doubt as to whether bringing user testing in-house is a shrewd move. It certainly sounds like a good idea, and many e-commerce managers we know would surely drool at the prospect of their very own testing lab, but is it the smart thing to do?
Other issues surrounding usability, user-experience and accessibility discussed in the briefing include the increased focus on usability as a digital service, differentiating usability and user experience, the cross-over between usability and other digital areas, using analytics to help drive usability and embracing social media and Web 2.0 to enhance user experience. The briefing is available free to all membership levels.
Recommended reports
Usability and User Experience Trends Briefing January 2009
A free, 12-page briefing with the latest trends and strategies related to user experience. Download this free report to learn about the importance of usability testing, the role of analytics, and how to use Web 2.0 to enhance user experience.
User Experience Buyer's Guide 2011
An invaluable resource for those investigating the market for user experience agencies, with profiles of 27 suppliers, the latest market trends and tips and pitfalls for buyers. The 263-report covers issues and trends affecting this sector, and contains information about best practice, as well as useful advice for those seeking a suitable supplier.
Head of Usability & Conversion at PRWD
4:53PM on 8th January 2009
Thanks for raising this debate Jake, its one which deserves the exposure both the trends briefing document and this article has provided.
An important consideration for businesses is how affective ‘guerrilla’ style user testing can be used in helping to identify potential usability barriers with their websites and IT systems. Rather than a business investing in user testing labs and therefore taking on the associated costs you would expect in a professional set-up, if they adopt more primitive methods of capturing feedback from users (ie from their own staff, especially if their business is a large organisation where staff from outside of the project team can be used) this can provide valuable insights into some of the more major usability barriers that may exist.
As you have pointed out, when a company is looking for a more detailed analysis of how their customers use and interact with their website, utilising the services of expert usability practitioners, although more expensive in the long run, does expose greater levels of actionable insights.
Especially for retailers, these insights will most often deliver increases to the likes of conversion rates, shopping basket sizes and number of returning visitors, and its through measuring the financial impact of these improvements over a 3, 6 and 12 month period that the business case for out-sourcing can be rubber-stamped.
One of your other key points you raised on carrying out usability testing in an un-biased environment is another primary consideration for businesses wanting to get the most honest user feedback on their site – the true value in user testing can be discovered when the end users taking part feel completely comfortable in being critical of the website/system they are assessing – unless it delivers a delightful user experience of course!
On a final note, one particular area of user testing which benefits from being facilitated by a usability practitioner is gaining tangible verbal feedback from the user during them carrying out an task on the website in question. Think out loud methodology, in particular ensuring a user shares their ‘expectations’ prior to clicking a particular link or image, are skills which can’t just be gained by reading a few articles and ‘how-to’ guides unfortunately!
If this helps readers of this article I have previously posted an article on ‘the business benefits of user testing’ and it includes pointers on ‘what to do if you don’t have a budget for user testing’, which may well be the case for some businesses, especially in the current financial climate.
Senior Research Analyst at Econsultancy
6:01PM on 8th January 2009
Thanks for the comments, Paul.
10:03AM on 15th January 2009
Hi,
I agree with Paul here on the benefits of in-house testing, even when this is 'quick and dirty' with staff, friends or family.
I've used a webcam with laptop and cheap software (total cost £80) to get very rapid feedback on prototypes and pre-production systems. The return on investment was pretty awesome as running an external test would have cost several K.
I think that the only way to gain experience of moderating tests is to actually perform them. I regularly give people copies of 'Don't make me think' by Steve Krug as this does a good job of explaining why some testing is better than no testing at all.
Another option for people is to go for the middle ground and get some training on moderation by an experience usability practitioner, build some practical experience through testing 'low budget' style and then hire a usability lab and run the tests yourselves. We found that this was a good balance between the two.
C.
Head of Usability & Conversion at PRWD
11:15AM on 15th January 2009
Hi Craig,
I'm a customer of your business LoveFilm and also a big fan of the overall user experience and engagement the site provides. It looks likes your balance of running your own tests by hiring labs is working really well and paying off in your customer experience, and as you say this combination helps keep the costs down.
The option of also getting training on user testing moderation by an experienced usability practitioner makes alot of business sense, and along with the training on e-commerce usability and best practice I am delivering for Econsultancy, providing user testing training is something we're also planning.
Keep up the good work at LoveFilm!