Phase 2: piloting and evaluating

Now that you have learned how we create an item list, we will show you how we evaluate whether it stands our requirements and what we still may have to revise. The idea behind pre-testing or piloting a test is that you take a test run first. Instead of going straight to your patient group, you pilot your test first in a healthy population. This is crucial to avoid wasting patients’ time with an unfinished test. We first want to know whether healthy speakers give the desired answers. In a small group, you will run your tests under the same conditions as you would run your test with the patients. In this way, you can evaluate what happens. Is the test too long?

Maybe it’s even too exhausting for a healthy person, let alone for the patient group. Does concentration last that long? Is the set up correct? Do you need to move to screen? The recording equipment? Is it comfortable for the participant? And even if you plan very carefully, you need to expect unexpected events. Piloting is therefore crucial to run into those problems before the actual testing. And most importantly, the piloting helps you to answer the question, do the items work as intended? Once the results are in, you can revise your list of items. Do some of them need to be omitted because they could not be named fast enough, accurately enough? Do some pictures need to be redrawn?

As a rule of thumb, consider to only keep items in the final version if the picture was eliciting the expected answer in 90% of the cases. In a clinical setup, the test will then go to the next step, standardising your paradigm in an even larger, healthy population of participants of both genders with a similar age and education, with the, in our case, brain tumour patients.