Strengths and weaknesses of conformance testing
The combination of automatic testing and manual testing takes advantage of the strengths of both methods.
The biggest advantages of automatic testing lie in its speed and scalability. It can quickly check a large web site with many pages on different browsers, operating systems and devices. It also limits human errors, for example, forgetting to check some links or criteria.
During website development, developers can use automatic testing to check for possible problems in the web content design. However, automatic testing can only find accessibility problems that can be found programmatically e.g. if alternative text for a multimedia content exists, if the colour contrast meets the requirements or if a table has a header or if a text input field in a form has a label.
Manual testing makes it possible to find accessibility problems that involve human understanding of the content. For example, an automatic testing tool can determine whether multimedia content has a descriptive text associated with it, but manual testing can find out whether the text provides enough information about the multimedia content.
In addition, manual testing can test whether assistive technology can access the web content. For large web sites, manual testing requires systematic processes and methods in order to cover all the pages and elements. The accuracy of manual testing depends on the evaluator’s knowledge of principles and guidelines and understanding of source code.
The weakness of conformance testing is that it misses user’s perspective.
Neither automatic testing nor manual testing involves real users.
Therefore they are not able to cover the users’ real needs and requirements. Research by Power et al from York University Guidelines are Only Half of the Story: Accessibility Problems Encountered by Blind Users on the Web has shown that only a portion of the accessibility problems encountered by real users is covered by the success criteria in WCAG.
© This text is a derivative of a work created by Høgskolen i Oslo og Akershus, and licensed under CC-BY BY 4.0 International Licence adapted and used by the University of Southampton. Erasmus + MOOCs for Accessibility Partnership.