I’m sure we could add plenty more to this article but I think it paints a relatively good picture that demonstrates why automated tools should be used with A LOT of care – especially those that claim to validate anything other than code validity.

I'm guessing that this article was in response to SiteMorse's PR piece. I'm also assuming SiteMorse will read this article, so I'd like to ask them to confirm which Web Content Accessibility Guidelines (WCAG1.0) their tool can categorically validate for conformance purposes. 

Trenton, I wouldn't recommend Bobby.

SiteMorse?

Thanks
Paul

On 13:29:34 12 August 2005 Trenton wrote:

An automated accessibility tool is a piece of software which can test a web page, or even an entire website, for accessibility. Automated accessibility tools are useful because they can save you a huge amount of time. Don't want to check images for alt text on each and every page on your website? Run the site through an automated tester and it'll do it all for you!

Automated accessibility testing tools have been around for a long time and have historically been a useful way of checking websites for accessibility. Bobby, one of the first and most well-known automated accessibility testing tools, is now almost 10 years, and although is no longer freely available, plenty of other free tools such as WebXact and Wave do exist.

But are these tools a little too good to be true? Can you test a website for accessibility so easily? Unfortunately the answer is a resounding no. There are a number of underlying problems associated with using just automated tools to test for accessibility:

Literal interpretation of guidelines

Any automated accessibility testing tool, being a piece of software, doesn't have very much in the way of common sense. It will interpret each and every accessibility guideline literally, without bearing any other thought to what else is on the page.

The definition of the word guideline, according to Dictionary.com, is “a rule or principle that provides guidance to appropriate behaviour”. A guideline simply offers guidance to what the best practice is - it shouldn't just be applied without regard to other factors.

For example, one of the W3C accessibility guidelines states that a table summary should be provided for all tables. (This summary doesn't appear on the screen, but it's read aloud to screen reader users before reading through the table content.) Table summaries are useful as they tell screen reader users what to expect in the table. However, there may be a heading directly before the table and it describes what the table is about. In this instance, this summary is essentially useless as it will just repeat what the previous heading said.

Can't check any content issues

The way that content is structured both on the page and across the website is a massive part of accessibility. A website may be perfectly coded and conform to the highest coding standards. If its content is poorly structured though, the site will prove difficult to impossible for some special needs web users.

There are a number of important accessible content considerations, none of which automated accessibility testing tools can check for. Some of these important considerations include:

- Front-loading content so that each paragraph begins with the conclusion
- Ensuring content has been broken down into manageable chunks with descriptive sub-headings
- Using lists wherever appropriate
- Ensuring that plain and simple language is used

Can't check many coding issues

The vast number of accessibility guidelines tend to be related to how the site is coded. Automated accessibility testing tools are unfortunately unable to test for many of these too. Examples of HTML-related accessibility considerations which these tools can't check for include:

- Ensuring that text is real text and isn't embedded within images
- Making sure that the site functions without the use of JavaScript or Flash
- Providing equivalent text links if using server-side image maps
- Ensuring that the structure within the HTML reflects the visual appearance (e.g. headings are labelled as headings within the HTML code)

Outdated guidelines are used

Automated accessibility testing tools generally use the W3C accessibility guidelines, which by now are over five years old. As such, a number of these guidelines are outdated and don't apply anymore. In fact, some of them are now thought to hinder accessibility rather than help, so it's best to totally ignore these guidelines.

For example, an automated accessibility testing tool will probably insist that form items contain default place holding text. It may also insist that links need to be separated by non-link text. Neither of these guidelines are relevant anymore and their implementation could make accessibility worse rather than better.

Most guidelines aren't properly checked

Automated accessibility tools can check for a number of guidelines, and can tell you when a guideline isn't being adhered to. However, when the tool claims that a guideline is being fulfilled this may in fact be a false truth.

For example, if all images contain alt text then the software will report a pass for this guideline. But what if the alt text isn't descriptive of its image? What if alt text is crammed full of nonsensical keywords for search engines? How can an automated accessibility tool possibly know this?

Warnings may be misinterpreted

The reports generated by automated accessibility tools provide warnings, as well as errors. These warnings are basically guidelines that the automated tool can't check for, but which may be errors. Often they're not, and in fact they're often not even relevant. However, some people reading a report may try to get rid of these warning messages by making the appropriate changes to their site. By doing so, they may be implementing guidelines that needn't be implemented and inadvertently lowering the website's accessibility.

Conclusion

Automated accessibility testing tools can be useful as they can save a large amount of time in performing some very basic checks for accessibility. However, they must be used with caution and they cannot be used as a stand-alone guide for accessibility checking. Indeed, some expert accessibility knowledge should always be applied in evaluating a site accessibility, perhaps in conjunction with the fantastic web accessibility toolbar to help dramatically speed up manual checks. 

Trenton Moss, Webcredible

While I agree it's a good idea to highlight the level of human interaction required to properly validate websites for accessibility, I do not think it is fair to call the inability of automated tools to detect content or coding issues a 'problem'.

Current tools like AccVerify do state pretty clearly there's a lot you need to check yourself, such as the content and coding issues you point out.

Also, calling the W3C guidelines completely  irrelevant is a little harsh. They form the basis for all modern accessibility testing and a lot of the guidelines continue to be very useful. The fact they are out of date is unfortunate - though according to the W3C the new guidelines should be released by the first quarter of 2006.

So I guess I'm agreeing with your last point: this can be a pretty confusing area where simple programs cannot answer all your needs. Expert consultancy is really the only way to properly ensure a site is and remains accessible.

There's no magic pill after all.

Hi Dan/Simon,

I’m delighted to see you embrace accessibility with such passion, if only everyone else did the same. I agree with the points you both made and I’m even impressed that one of you claim to be an expert on the use of RDF in your profile – this is a rare commodity.

I don’t see where Trenton says that the WAI guidelines should be dismissed or that ‘using automated tools is a problem’ – this is taken out of context IMHO. He might have given his opinion regarding the ‘relevance’ of ‘some’ guidelines, but this isn’t the same as saying the entire initiative should be dismissed. In fact, Trenton’s article(s) demonstrate to me that he endorses accessibility and understands how to implement the WAI guidelines better than most who claim to be ‘usability and accessibility’ experts – albeit with a bias towards the use of CSS. This isn’t a bad thing as the industry needs people to say what they think they’re experts in and not try to say they specialise in everything. I could expand on some of his views and perhaps argue some minor points, but that’s trivial in the scheme of things.

The main point that I took from his article and what I’m in total agreement with as a member of the W3C Evaluation & Repair Tools working group, is that there are a number of underlying problems associated with using just automated tools to test for accessibility.

All too often I see a Bobby logo to demonstrate compliance with the WAI guidelines. Some companies actually include ‘Bobby compliance’ as part of their portfolio of services – these companies must not be engaged if they use Bobby or any other tool as their ONLY means to validate a Web site’s compliance with the WAI guidelines. Otherwise, organisations that use them could be left open to potential litigation and more importantly, damage to their brand when major flaws go undetected. It is widely accepted by the W3C that there is no tool on the market that can validate an entire Web site for compliance with its guidelines. Nor will there ever be a tool that validates all of the WAI guidelines.

Whether I agree with Trenton’s specific reference to the relevance of some guidelines isn’t really important. I discuss (debate) the guidelines within the W3C where we are responsible for assisting with the revision of WCAG2.0 to make them less ambiguous, easier to understand and easier to validate. So if the comment “these people infer that the guidelines were of no value” was referring to me, I’m sure you will change your view.

Automated tools should be used wherever possible as it reduces the overall time to validate a Web site. We use validators and analysers frequently as they save us a lot of time. In fact, we provide our Client’s agencies with lots of goodies to assist them in getting it right before passing anything onto us for certification. This allows the agencies to not only design and build compelling accessible solutions, but also to test it as much as their capabilities permit, thereby reducing the amount of iterations required before going live.

I’m assuming the comment regarding the promotion of services was directed at me. This is an insult if you think I was trying to indirectly endorse our services. If I was to do that, I’d link to our all news releases and stories written about us in the WAI news letter, NMA, Marketing Week… ;) My teams dedicate a very large portion of their time to the creation and revision of Web standards and technologies that people take for granted today – the majority of this goes unnoticed from our brands perspective (the W3C Mobile Web Initiative is an exception). The objective of my note was to highlight the necessity to validate work done by someone who claims to build an accessible Web site. It is widely accepted within the ‘software’ industry (including the Web) that all development work should be tested to minimise the risk of errors going undetected. A developer checking their own homework is never advisable – developers who work for me would always release code with defects if it wasn’t for our testers (in 10 years working with the Web, I’ve never met a developer who wouldn’t). BTW, we do not design or build Web sites for clients.

My reference to user testing was a qualified statement given that the DRC reported that although the WAI guidelines are accepted as the de facto guidelines for accessibility, “they are not a substitute for user testing by disabled people”. This is not to say that every project must have user testing by disabled people, as that just wouldn’t be feasible for most organisations and they could turn their back on accessibility altogether if they thought it was essential in every instance. We should not assume to know how people with certain disabilities use assistive technology or to know all of their preferred ways of surfing Web sites, so its good practise to incorporate it wherever possible. This is the very reason you have beta testing within the software industry – users will usually use software in a way you couldn’t imagine, so they will find errors that could otherwise go unnoticed by an experienced tester.

You could always get the RNIB on the case and pay them a whacking £100 per hour for the privilege, or AbilityNet who are even more expensive; a disabled tester at £135 per hour! This is the type of costing model that puts many off independent accessibility assessments. It certainly doesn’t help smaller organisations who want to conform but can’t afford it, when the only assessors they are aware of are those who raise brand awareness through threatening people with court action and releasing ‘industry reports’. If the RNIB want to actually help the industry to embrace accessibility for the blind (which is their primary focus), then they should make their services more affordable and in line with profit making organisations who provide a similar service. Threatening court action on one side and then offering an expensive consultancy service on the other is questionable in my personal opinion, but this is for another day’s story that would get most people hot under the collar.

I don’t think anyone said that user testing replaces the WAI guidelines as that wouldn’t make sense. User testing is complimentary to WAI conformance validation.

Please take this note in the spirit in which it is intended – impartial/independent, helpful and with a smile.

That’s all for now folks :)
Paul

On 15:28:04 18 August 2005 Dan Zambonini wrote:

I couldn't agree more about defending the WAI guidelines. 

I was at a roundtable last year, with bigwigs from major companies and accessibility 'experts'.  The discussion, guided mainly by the accessibility experts (including well known voices within that domain), centered on the absolute dismissal of the guidelines, and how a company could not even attempt to make an accessible site unless accessibility consultants/user group testing was employed.

The accessibility guidelines may have their limitations, but I truly believe that roundtables of this nature wouldn't even exist without the production and promotion of those guidelines in the web industry.

It's fair enough to promote your services, to discuss the benefits of user testing, and to point out the limitations of the guidelines.  But for these people infer that the guidelines were of no value?  Absolutely not.  I guess they don't understand the world outside the cash-rich enterprise, where (unfortunately) 99.995% of people/organisations on the web don't have the cash to spend on user testing and external consultants.  And for this, the web guidelines are a perfect starting point.

Right, I need to go calm down now...

Thanks for clearing that up Dan :) Most threads within the WAI and MWI discussions tend to go on forever and a day as one small point can be misinterpreted. I almost feel obliged to caveat everything I say in order to eliminate the need to reiterate the same stuff in different ways, hence the long response... :)

Paul

On 09:28:16 19 August 2005 Dan Zambonini wrote:

Hi,

(Just a quick follow up...) - Wow, long posting!  I hadn't actually read your original post, I was just replying to the one above mine - apologies if anything I said sounded as if it made negative reference to your post... I wasn't commenting on your services (having not read your post), but was commenting on the people at the roundtable (hence the "But for _these people_ infer that the guidelines were of no value").  Just thought I'd clear that up!

Ta,

Dan

User testing doesn't have to be that expensive though! Their costs don't do much to encourage usability companies, agencies and their clients to embrace the concept of accessibility certification (with or without user testing) as they assume everyone charges the same as the RNIB and AbilityNet.

Paul

On 10:17:22 19 August 2005 studio24 wrote:

Fair comments Paul, I think we're all agreeing in principal. I was just responding to how I read the original post.

It's true accessibility guidelines are up for enormous debate and interpretation, I only hope the next version of the Web Accessibility Guidelines offer a clearer framework for accessibility testers.

User testing is all very well, but as you point out unfortunately very expensive for most companies. Also bear in mind most companies still pay token respect to accessibility so are not prepared to fork out payment for additional testing, instead expecting the web agency as the expert to know everything.

So most of us end up relying on shared knowledge throughout the community, and when it comes to new techniques (i.e. AJAX) it's down to educated guesswork.

It is fine to install screen readers ourselves and use other tools to try and second guess how a disabled user might use a website, but I agree you can't really beat letting real users test a site.

I was at the Web Standards (@media) conference recently and one of the most illuminating speakers was Robin Christopherson from AbilityNet who actually walked us through how he (as a blind user) uses a website.

It's good to see accessibility promoting debate though.

simon