Skip to 0 minutes and 1 secondINTERVIEWER: Do you also have the possibility to do user testing and how much do you rely on expert and automatic testing?

Skip to 0 minutes and 9 secondsANDRE MEIXNER: Mm-hmm.

Skip to 0 minutes and 13 secondsFor sure, we're doing many user tests but in terms of accessibility, not so much because most accessibility problems are technology-based. There are incompatibilities between, for example, the screen reading software and the application. So to find the accessibility issues, we are doing mostly expert tests to see if they are compatible and give the programming instructions to developers to fix these problems. And then if the accessibility problems in terms of technologies are fixed, then we are using a user test to find out if the application meets the needs of the users. And on that basis, it totally doesn't matter if you have a disability or not. Then it's just usability. So we are doing these two things separated.

Skip to 1 minute and 12 secondsAnd your question to the automated tests-- we are doing automated accessibility tests and for that, we have an automation expert team in our test centre. And together with this team, we've developed a website crawler, for example, that can crawl to all the different dialog masks of a website and find and list all the problems in terms of accessibility, for example, if some alternative text to pictures are missing or if headline texts are missing that the screen reader needs to structure the website and so on. And yeah, we've developed this and we are combining these automated tests with our expert tests to give the developers even more detailed information where to find the things that they need to fix.

Skip to 2 minutes and 8 secondsINTERVIEWER: What do your customers expect from your department to fix accessibility issues?

Skip to 2 minutes and 16 secondsANDRE MEIXNER: From our perspective, we ourselves, we are only doing the quality part. So we are just doing the testing. We are not doing the development. And for me, it's really important to separate these things completely. So from our experience, it's a big problem if the developer is testing his or her own programmed software because she won't find all the problems. So our approach is we are testing the software and we try to empower the developers to fix the problems in their development. And for that reason, as I said before, because of our many, many testings we are doing, we can provide even code-based solution information to the specific framework that is used, for example.

Skip to 3 minutes and 9 secondsAnd so we are the ones that can be asked if the developers have problems.

Skip to 3 minutes and 19 secondsINTERVIEWER: Thank you very much.

Skip to 3 minutes and 20 secondsANDRE MEIXNER: Thank you.

What does user testing involve?

Involving users in accessibility tests is compatible with usability testing. In fact, sequence and evaluation methods are applicable to both types of testing.

The main difference is, accessibility testing involves users with disabilities and usability testing tends to overlook this aspect of testing. However, usability testing with people who have disabilities often exposes both types of problems.

In this video, Andre Meixner talks about this and about the relationship between expert testing, user testing and automatic testing.

accessibility as a subset of Usability

© This image is created by Technische Universität Dresden and licensed under CC-BY BY 4.0 International Licence. Erasmus + MOOCs for Accessibility Partnership.

When conducting accessibility tests with users, we need to answer the following questions:

  1. What do I want to test?
  2. With whom should I test?
  3. How many participants do I need?
  4. Which tasks should be part of the evaluation?
  5. Which evaluation methods suits best?

1. What do you want to test?

Results from the conformance test are a starting point to find out which accessibility issues should be further investigated with real users. Additionally, your experiences with assistive technologies is an indicator as to what you want to test with end users.

For instance, you might not be experienced enough to test whether forms or CAPTCHAs are accessible for people using a screen reader. In these cases, it will be beneficial to invite end user for user tests.

2. With whom should I test?

On the one hand, you should focus on target accessibility issues that require user feedback. For instance, if you want to check if a slideshow is navigable by screen readers, one target group would be blind screen reader users. There is an added benefit of also focussing on the user audience of the web site. If you are going to evaluate an e-learning platform designed for those with specific reading difficulties, you should involve those who have dyslexia.

It is also important to consider the skills of the target audience of your web site making sure novice, average, and power users are considered.

3. How many participants do I need?

There are varying arguments about how many participant are needed to identify problems. Nielsen Norman recommend that 5 users should be sufficient for qualitative user studies (with the exception of eye tracking).

As people with disabilities may use web sites in a significantly different way depending on the assistive technologies they use and their user needs, it is important to include a range of users with different disabilities.

4. Which tasks should be part of the evaluation?

Don’t ask users questions such as ‘Do you think this form is accessible?’ Instead, create smaller tasks around an accessibility issues, for instance: ‘Go to the home page and create an account’ (including the page with the form for which you want to receive feedback).

Selection of tasks should correspond to the main functionalities of the web site and reflect accessibility issues that are of interest.

5. Which evaluation methods suits best?

There are a broad variety of evaluation methods to involve users in the development of web sites. Most of them are developed for usability testing.

However, Mei et al in their paper ‘Contrasting usability evaluation methods with blind users’ have shown, involving people with disabilities in user-centred design requires some adaptations to traditional evaluation methods, for instance, tactile paper-prototyping for blind users (see image below), or to consider advantages and disadvantages of using local and synchronous remote tests.

tactile paper prototype

© This image is created by Technische Universität Dresden and licensed under CC-BY BY 4.0 International Licence. Erasmus + MOOCs for Accessibility Partnership.

What do you think might be the strengths and weaknesses of user testing?

© This video is created by Technische Universität Dresden and licensed under CC-BY BY 4.0 International Licence. Erasmus + MOOCs for Accessibility Partnership.

© This text is a derivative of a work created by Technische Universität Dresden, and licensed under CC-BY BY 4.0 International Licence adapted and used by the University of Southampton. Erasmus + MOOCs for Accessibility Partnership.

Share this video:

This video is from the free online course:

Digital Accessibility: Enabling Participation in the Information Society

University of Southampton