Testing for Digital Accessibility
An apology: this is one of the longest pages on the site, and I ran out of time to past in all the hyperlinks and style the page. You should be able to find the references by web searching for the title and author’s name. – Joshua Randall
How to evaluate your website or app to ensure it is accessible.
Testing Approach
Karl Groves, an accessibility expert and founder of the company Tenon, make the provocative statement that “Current automatic accessibility testing practices take place at the wrong place and wrong time and is done by the wrong people.” Groves continues: “Testing must happen before deployment and before content is published.”
- Designers who create mockups should test their work before giving it to developers to implement
- Developers should test their work before it is submitted to version control
- Content authors should test their work before publishing
- QA staff should run acceptance tests using assistive technologies
- UX staff should do usability tests with people with disabilities
Read more in ‘Everything you know about accessibility testing is wrong (part 1)’, part 2, part 3, and part 4.
This should not discourage you from testing at all. Some testing is better than no testing!
Choosing a tool
Karl Groves of Tenon wrote an article on choosing an automated accessibility testing tool (2013, January 28).
AccessibleTech.org has a similar article titled How can I select a Web accessibility software tool?
Marcy Sutton, an Angular core team member, has a presentation on the magic of automated accessibility testing (2015).
CSS Tricks covers a few tools in an article by Chris Coyier (2017, December 14).
Testing advice
- eBay has the cleverly named OATMEAL (Open Accessibility Testing Method for Experts And Layfolk) that goes through how they do their testing.
- Vox Media publishes a checklist for QA teams. It’s basic but a decent starting point.
- Duke University testing checklist (PDF, 87 KB, 6 pages).
Business Disability Forum’s advice on User Testing
User Experience Testing
A policy regarding User Experience Testing for accessibility ensures that products and services delivered by the IT organisation meet acceptable criteria for accessibility rather than finding out from end users. The policy must state how accessibility fits into the testing framework. This includes which standards or guidelines the testing is performed against, how any failures are to be dealt with and how testing is actually to be performed.
End User Testing
When in project planning stage, you might want to consider user testing for Accessible Technology (AT) – including text to speech, speech to text, screen magnifiers and other frequently used technology. It may be helpful for you to identify what AT you already have within your organisation.
User testing for AT could be carried out by employees who are already familiar with the software and may be able to advise you on what improvements need to be made.
Testing software before it is rolled out through your organisation will potentially save you money and ensure that you are providing an equal playing field for all your users.
User testing should be considered for both internal and external projects.
Testing Tools
Screen Readers
Web AIM (Web Accessibility In Mind) published an article on the benefits of testing web content with screen readers. (last updated in 2013, but still relevant today)
Please refer to the Screen Readers and Assistive Technology section of this website for additional information on the various tools available.
The recommended screen reader and browser combinations are
- Voiceover with iOS
- NVDA with Firefox
- JAWS with Internet Explorer
WAVE
WAVE, the Web Accessibility Evaluation tool, is produced by the organization WebAIM (Web Accessibility In Mind). WAVE is available as a webpage into which you can input a single URL, or a browser plugin for both Chrome and Firefox.
There’s an API version as well: http://wave.webaim.org/api/details
WAVE is one of the de facto standard tools for accessibility evaluation, but it’s important to understand WAVE has limitations, as explained in the following extensive advice from accessibility expert Kurt Bunge at Interactive Accessibility.
Typically speaking, WAVE errors are WCAG violations based on the filter you’ve chosen. However, some of these can be somewhat deceiving because the tool is only a pre-programmed tool and it can produce false positives. For example, the tool can tell if an image has a text alternative, but it cannot tell if the text alternative is appropriate for the image. It also cannot tell the difference between images that are redundant or decorative and should therefor be ignored by assistive technology versus images that should be interpreted.
Alerts in WAVE are usually usability issues for those who use assistive technology and not WCAG violations. The title element used on link is a perfect example. This is not a WCAG violation but can affect a screen reader user’s experience. These types of issues are reported as “Low” severity issues in your accessibility audit document. However, many alerts need to be verified with some form of code inspection depending on what the issue is. For example, if the link text and title attribute contain the same value, that is a usability issue but not a WCAG violation. If the title attribute contains text that adds additional, helpful information to a link, it is not a usability issue. So again, some manual inspection of the code would need to be done. When running the links test using our tool, these are reported as warnings, but we also output the link in question so you can see right away if the title text is redundant to the link text or not.
Keep in mind that automated tools only catch about 25% of real accessibility issues and most of these tools rely on a certain level of accessibility knowledge of those who use them. It is pretty difficult to rely completely on an automated tool to determine your level of compliance, but they can be good spot-checkers. I would never rely 100% on any automated tool to tell me if I am in complete WCAG compliance.
(Source: email message, 2018 April 4.)
the W3C validator and why you should care
(in progress) already described on another page… it’s for validating HTML and finding WCAG errors on a page-by-page basis, not an automated tool. TODO: write a section explaining step-by-step how to use it, and how to use the Nu version with the plug-in to look only for WCAG violations.
Consulting Firm Tools
All of the big accessibility consulting firms offer some kind of testing tool.
Deque – aXe
aXe is available as a browser plug-in similar to WAVE, based on a rules engine.
There is also aXe-Core which can parse raw HTML. The tool itself is open source but Deque also sells a more powerful commercial version.
Interactive Accessibility – IA Toolkit
The IA Toolkit is a browser extension like WAVE but turned up to 11.
Level Access – AMP
Accessibility Management Platform (AMP) is a web-based platform for testing.
The Paciello Group – ARC
TPG’s Accessibility Resource Center (ARC) is an automated tools that “looks for machine-detectable accessibility defects within automated content.”
Tenon
The company’s tool, also named Tenon, extensively documents its API.
You can sign up for a free trial: https://tenon.io/register.php
Get the code: https://tenon.io/getcode.php
Training videos: https://tenon.io/documentation/videos/
Mallet is an affiliated tool that can “Automatically fix over a dozen different types of issues out of the box, Support creation of custom issue fixes, and Generate reports on additional types of code that need manual inspection.”
Other Tools
Many, many other testing tools are available. Here are a couple.
Pa11y is an open-source tool that can evaluate web pages and is also available as a webservice and in a command-line interface.
ember-a11y-testing – Some kind of testing framework, not fully evaluated yet.
Inclusive Usability Testing
When you do usability testing with real humans, you will discover things that you never could with automated tools.
User Research with People with Disabilities (Word document, 2.8 Mb, 11 pages). David Sloan. (2018, April 25). Sloan, the user research lead for The Paciello Group (TPG), explains how you can effectively conduct research with people with disabilities.
Inclusive Usability Testing (Slideshare, 68 slides). Adrian Roselli. (2018, April). Covers the additional setup and procedures you will need for usability testing with people with disabilities.
Tips for Conducting Usability Studies with Participants with Disabilities. Peter McNally. (2018, March 12). Smashing Magazine. He shares his lessons learned from real-world usability testing.
Mobile Testing
(This section is still a work in progress.)
The Paciello Group’s Mobile Testing Guide for Android and iOS (PDF, 2.5 Mb, 31 pages). (2017, April). Based on iOS 9 and Android 4.4.