CAPTCHAs — those computer-generated images commonly used with website forms that challenge users to prove they’re human — are a popular tool in the arsenal against web spam.

But when looked at from a cost-benefit analysis standpoint, do they help or hurt conversions?

A great post on the SEOmoz blog details how one person set out to answer the question. ‘chenry‘ of the SEOmoz community conducting a six-month study across 50 websites that he owns or has access to. For the first three months of the study, half of these websites implemented CAPTCHA on their forms and the other half didn’t. For the last three months, the settings were reversed.

The results:

  • With CAPTCHA enabled, there was an 88% reduction in spam.
  • With CAPTCHA disabled, 4.1% of the conversions recorded were spam.
  • With CAPTCHA enabled, 7.3% of the conversions were either detected as spam or failed.

The analysis: CAPTCHAs do reduce spam conversions but it also seems to significantly increase the number of conversions that fail.

While further study would be useful to quantify exactly how many legitimate conversions fail because of CAPTCHA, the implication here is that the use of CAPTCHAs should be evaluated carefully on a case-by-case basis. If you’re using CAPTCHAs on your websites, it probably makes sense to consider whether the pros outweigh the cons.

In the case of ‘chenry‘ and his websites, when CAPTCHA was disabled, there were 2,134 total conversions. Of those, 91 were spam. While CAPTCHA reduced the spam conversions to only 11, were the 159 total failed conversions that CAPTCHA caused a smart trade-off? In other words, would you rather deal with 91 spam conversions or lose 159 conversions, some of which were almost certainly legitimate? Obviously, the value of each conversion will play a big role in answering this.

While ‘chenry‘ mentions that he’s now avoiding traditional CAPTCHAs and is instead experimenting with the Honeypot” CAPTCHA technique, you don’t necessarily need to ditch CAPTCHAs. They can be effective, especially in high-volume environments in which the value of each form conversion is low.

Where you determine that CAPTCHAs can be of value, you can minimize failed legitimate conversions by:

  • Choosing a CAPTCHA implementation that is legible. This would seem like a common sense way of making sure your CAPTCHA doesn’t turn legitimate users away but chances are you’ve come across a CAPTCHA implementation that looked more like a Rorschach test than a CAPTCHA. Don’t make the same mistake.
  • Offer an audio CAPTCHA. This is a good idea anyway since graphic-based CAPTCHAs are problematic for visually impaired users but audio CAPTCHAs can also assist users who have trouble reading the letters/numbers they’ve been presented.
  • Make it easy to refresh. If for whatever reason someone can’t figure out a CAPTCHA, make it easy for them to try a new one, ideally without reloading the page completely.
  • Provide instructional copy. Be sure your users understand what a CAPTCHA is and how to complete it; provide some text that makes the process simple for users who don’t know what they’re looking at.
  • Add a first layer of defense. You don’t necessarily need to include a CAPTCHA on every form you display. You can use CAPTCHA selectively when a first layer of filters flags a suspicious conversion. For instance, you could require a CAPTCHA after a form has been submitted and the expected referrer is not present or when a form is loaded by a user with an IP address from geographic regions that are not typical users of your website.

Photo credit: cogdogblog via Flickr.