1. Trenton Moss

    Director at Webcredible

    12 January 2007 15:57pm

    Trenton Moss

    There are currently three different options available to you when testing a website for accessibility:

    • Accessibility audit - An accessibility expert reviews your site, highlighting any accessibility issue
    • Accessibility testing - Real disabled users complete common tasks on your website whilst a moderator notes all problems they experience
    • Automated accessibility testing - An automated program evaluates your website against accessibility guidelines

    Accessibility audit

    An accessibility audit involves an accessibility expert reviewing your site, highlighting all accessibility issues and providing recommendations for fixing them. The reviewer would typically use assistive software used by disabled web users (e.g. a screen reader) to effectively carry out the audit, along with the Web Accessibility Toolbar.

    You could hire an external accessibility consultancy to do this, but it's also possible to conduct the audit in-house. By reading through the W3C accessibility guidelines and attending a web accessibility training course (the latter to learn 'real-world' accessibility) you can gain a base level of accessibility knowledge. You should then be able to get your website up to a reasonable level of accessibility.

    The main benefit of using an accessibility audit to evaluate your website is that accessibility audits are significantly cheaper and quicker than accessibility testing. Accessibility audits are often more comprehensive than accessibility testing in their depth and breadth of recommendations.

    The main disadvantage of accessibility audits is that they're not designed for knowledge transfer. As such, your web team won't gain a great understanding of web accessibility nor are they likely to get much extra buy-in into accessibility. Both of these can be remedied through effective accessibility training.

    Accessibility testing

    There are a number of organisations in the accessibility world that swear by accessibility testing. The Disability Rights Commission, for example, have consistently said that testing a website with real disabled users is the only way to ensure it offers optimum accessibility. Their formal investigation into the accessibility of 1000 websites and the PAS 78 document they spearheaded both very strongly state that accessibility testing is the way to uncover all accessibility issues.

    The main benefits of conducting accessibility testing include:

    • You'll uncover website usability issues too
      As accessibility testing involves testing your website with real people, you'll get the side benefit of uncovering website usability issues too. If however you're going to invest the time and money involved in conducing accessibility testing then you might as well just carry out regular usability testing11. Regular usability testing will uncover more usability issues as disabled users tend to take longer to complete tasks so carry out fewer tasks in a testing session.
    • It's a great learning experience for your web team
      Running accessibility testing sessions, particularly if you're able to watch them live, is a great learning experience for everyone involved. There's nothing more eye opening than watching real users with real disabilities attempt to use your site. Accessibility testing is also a great way of getting the web team to get buy-in into accessibility.

    There are also some disadvantages of using accessibility testing as a way of evaluating your website's accessibility:

    • It's extremely expensive to do properly
      In order to carry out statistically valid accessibility testing you'll need to test with at least 5-6 users per user group. This means testing with 5-6 screen reader users, 5-6 screen magnifier users, 5-6 motor impaired users, 5-6 users with learning difficulties etc. Within these groups there are sub-groups (for example, there are different types of motor impairments) so to do proper accessibility testing you'll have to test with around 30 users.
    • It'll find fewer problems than an accessibility audit
      Surprisingly, accessibility testing actually brings up fewer problems (and therefore fewer recommendations) than an accessibility audit. This is basically because real users often don't realise when a problem or nuisance occurs so don't voice their annoyance.

      For example, vertical bars are often used to separate horizontal navigation links, with each vertical bar being announced to screen reader users as “vertical bar”. Screen reader users are unlikely to comment or complain about this as they probably won't know what a 'vertical bar' is. An accessibility audit would however point this out as a problem and offer the solution (the solution would be to remove the vertical bars and insert them as right borders (through the CSS) on to the links).

    Which accessibility evaluation technique should you use?

    Generally speaking, it's not necessary to conduct accessibility testing most of the time. To conduct accessibility testing properly is extremely expensive and simply not worth the return on investment for most organisations. It's unlikely to highlight any major accessibility issues that don't come out of the accessibility audit (provided that the audit is carried out by an accessibility expert).

    Are we really recommending to not do accessibility testing then? Well mostly, yes. Accessibility is far more guideline-driven than usability so accessibility solutions can easily be transferred from one website to the next. Provided that some organisations continue to carry out accessibility testing, you can 'piggy-back' off the knowledge that they've gained. By employing an accessibility expert to carry out an accessibility audit you'll quite quickly have this 'real-world' accessibility knowledge applied to your website.

    If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented. There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit.

    In reality though, you often only need to conduct accessibility testing if you want your web team to learn about accessibility and gain buy-in into it. Accessibility testing can also be useful if you wish to test complex interactions (e.g. Flash, complex JavaScript) that can sometimes fall beyond the scope of an accessibility audit.

    But what about automated accessibility testing tools?

    Generally speaking, automated accessibility testing tools are very limited in being able to uncover real problems. They can't adequately check for the majority of accessibility guidelines and can sometimes report problems that aren't actually problems. For more on the disadvantages of automated testing tools, read our article, The problem with automated testing tools.

    The only real benefit of using an automated testing tool is to get a top level feel of how accessible (or inaccessible) your website is. In terms of getting real recommendations for fixing real problems then an accessibility audit is vastly superior.

    Trenton Moss, Webcredible

  2. Peter Collins Gold

    Director at WUP

    19 January 2007 10:02am

    Peter Collins

    This is a very interesting article and I agree with much of what Trenton Moss has said. I would however, like to take issue with one or two points he makes:

    Trenton says “The Disability Rights Commission… have consistently said that testing a website with real disabled users is the only [my italics] way to ensure it offers optimum accessibility… the PAS 78 document… very strongly state that accessibility testing is the way to uncover all [my italics] accessibility issues..”

    I am not an apologist for the DRC but that is not my reading of their position. PAS 78 says “Comprehensive evaluation of web site accessibility should involve a combination [my italics] of conformance with the technical requirements of WCAG, and user testing of accessibility features.” PAS 78 talks extensively about expert reviews and conformance inspection – why these are useful and how they should be done. I would also be surprised if they, or anyone else, would argue than any one approach or even any combination of approaches would identify all issues.

    As far as I can see they, and I, would not disagree with Trenton’s statement “If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented” – this is eminently sensible, as he says “There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit”

    However, I, (and I am sure the DRC) would not agree with his statement “[user testing is] unlikely to highlight any major accessibility issues that don't come out of the accessibility audit.” Having conducted lots of disabled user tests after thorough accessibility audits (done by very competent accessibility consultants) I can assure Trenton that “Oh yes it will!” For example in a recent disabled user testing session for a well know supermarket client we identified over 10 serious issues (issues that would prevent a user achieving their goals) not identified in the audit. One of these included not being able to find the ‘Add to basket’ button!

    A key reason why expert assessments fail to find all serious issues is the same whether it’s a usability or accessibility audit. The expert can never be expert enough! It is almost impossible for a reviewer to be fully aware of how the user thinks and will behave. How can a male, 30 something, HCI expert get inside the head of a blind, wheel chair bound woman of 40 with a motor neuron disease? Having watched many hundreds of people on web sites I am now less likely to predict how I think users will behave than earlier in my career. My view of the world is inevitably influenced by the fact I am male, white and 50+! The other benefit of testing with disabled users is that it identifies usability issues not identified by testing with able bodied users – but do effect able bodies users. It is my view, and clearly that of the DRC, that only by testing with disabled users, as well as undertaking an accessibility audit can you be sure you have identified the serious accessibility (and usability) issues.

    Peter Collins - Web Usability Partnership

Reply to this thread

Log in to reply to this thread or join Econsultancy for free so you can post to our forums along with other benefits.