Director at Webcredible
12 January 2007 15:57pm
There are currently three different options available to you when testing a website for accessibility:
An accessibility audit involves an accessibility expert reviewing your site, highlighting all accessibility issues and providing recommendations for fixing them. The reviewer would typically use assistive software used by disabled web users (e.g. a screen reader) to effectively carry out the audit, along with the Web Accessibility Toolbar.
You could hire an external accessibility consultancy to do this, but it's also possible to conduct the audit in-house. By reading through the W3C accessibility guidelines and attending a web accessibility training course (the latter to learn 'real-world' accessibility) you can gain a base level of accessibility knowledge. You should then be able to get your website up to a reasonable level of accessibility.
The main benefit of using an accessibility audit to evaluate your website is that accessibility audits are significantly cheaper and quicker than accessibility testing. Accessibility audits are often more comprehensive than accessibility testing in their depth and breadth of recommendations.
The main disadvantage of accessibility audits is that they're not designed for knowledge transfer. As such, your web team won't gain a great understanding of web accessibility nor are they likely to get much extra buy-in into accessibility. Both of these can be remedied through effective accessibility training.
There are a number of organisations in the accessibility world that swear by accessibility testing. The Disability Rights Commission, for example, have consistently said that testing a website with real disabled users is the only way to ensure it offers optimum accessibility. Their formal investigation into the accessibility of 1000 websites and the PAS 78 document they spearheaded both very strongly state that accessibility testing is the way to uncover all accessibility issues.
The main benefits of conducting accessibility testing include:
There are also some disadvantages of using accessibility testing as a way of evaluating your website's accessibility:
For example, vertical bars are often used to separate horizontal navigation links, with each vertical bar being announced to screen reader users as “vertical bar”. Screen reader users are unlikely to comment or complain about this as they probably won't know what a 'vertical bar' is. An accessibility audit would however point this out as a problem and offer the solution (the solution would be to remove the vertical bars and insert them as right borders (through the CSS) on to the links).
Which accessibility evaluation technique should you use?
Generally speaking, it's not necessary to conduct accessibility testing most of the time. To conduct accessibility testing properly is extremely expensive and simply not worth the return on investment for most organisations. It's unlikely to highlight any major accessibility issues that don't come out of the accessibility audit (provided that the audit is carried out by an accessibility expert).
Are we really recommending to not do accessibility testing then? Well mostly, yes. Accessibility is far more guideline-driven than usability so accessibility solutions can easily be transferred from one website to the next. Provided that some organisations continue to carry out accessibility testing, you can 'piggy-back' off the knowledge that they've gained. By employing an accessibility expert to carry out an accessibility audit you'll quite quickly have this 'real-world' accessibility knowledge applied to your website.
If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented. There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit.
But what about automated accessibility testing tools?
Generally speaking, automated accessibility testing tools are very limited in being able to uncover real problems. They can't adequately check for the majority of accessibility guidelines and can sometimes report problems that aren't actually problems. For more on the disadvantages of automated testing tools, read our article, The problem with automated testing tools.
The only real benefit of using an automated testing tool is to get a top level feel of how accessible (or inaccessible) your website is. In terms of getting real recommendations for fixing real problems then an accessibility audit is vastly superior.
Trenton Moss, Webcredible
Director at WUP
19 January 2007 10:02am
This is a very interesting article and I agree with much of what Trenton Moss has said. I would however, like to take issue with one or two points he makes:
Trenton says “The Disability Rights Commission… have consistently said that testing a website with real disabled users is the only [my italics] way to ensure it offers optimum accessibility… the PAS 78 document… very strongly state that accessibility testing is the way to uncover all [my italics] accessibility issues..”
I am not an apologist for the DRC but that is not my reading of their position. PAS 78 says “Comprehensive evaluation of web site accessibility should involve a combination [my italics] of conformance with the technical requirements of WCAG, and user testing of accessibility features.” PAS 78 talks extensively about expert reviews and conformance inspection – why these are useful and how they should be done. I would also be surprised if they, or anyone else, would argue than any one approach or even any combination of approaches would identify all issues.
As far as I can see they, and I, would not disagree with Trenton’s statement “If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented” – this is eminently sensible, as he says “There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit”
However, I, (and I am sure the DRC) would not agree with his statement “[user testing is] unlikely to highlight any major accessibility issues that don't come out of the accessibility audit.” Having conducted lots of disabled user tests after thorough accessibility audits (done by very competent accessibility consultants) I can assure Trenton that “Oh yes it will!” For example in a recent disabled user testing session for a well know supermarket client we identified over 10 serious issues (issues that would prevent a user achieving their goals) not identified in the audit. One of these included not being able to find the ‘Add to basket’ button!
A key reason why expert assessments fail to find all serious issues is the same whether it’s a usability or accessibility audit. The expert can never be expert enough! It is almost impossible for a reviewer to be fully aware of how the user thinks and will behave. How can a male, 30 something, HCI expert get inside the head of a blind, wheel chair bound woman of 40 with a motor neuron disease? Having watched many hundreds of people on web sites I am now less likely to predict how I think users will behave than earlier in my career. My view of the world is inevitably influenced by the fact I am male, white and 50+! The other benefit of testing with disabled users is that it identifies usability issues not identified by testing with able bodied users – but do effect able bodies users. It is my view, and clearly that of the DRC, that only by testing with disabled users, as well as undertaking an accessibility audit can you be sure you have identified the serious accessibility (and usability) issues.
Peter Collins - Web Usability Partnership
Free market research on digital marketing
Daily Pulse: award winning newsletter
It takes 30 seconds to register