Measuring the lived experiences of people with disabilities

In this step, we will be learning about how to measure the lived experience of people with disabilities in quantitative studies. Research indicators are what you use to measure the specific lived experiences that are of interest to your research question. For example, let’s say you’re doing a study on access to education, how would you measure access to education? In other words, what indicators would you use? Some examples of indicators of access to education include the proportion of children enrolled in school, the proportion of children completing primary school, or average score on tests. As we discussed in week 1, it’s important to set research objectives that follow the SMART guidance. The same guidance can also apply to choosing indicators.

As a reminder, SMART stands for Specific, Measurable, Achievable, Relevant, Timely. This means indicators should be selected so that they are specific to the research question and outcome of interest. Ask, do the indicators capture the outcome of interest in different groups, including people with disabilities? Your indicators should also be measurable. This means that your indicator is using data that is easily quantified or observed. For example, did you complete primary school? Yes or no. It’s a measurable indicator. In contrast, asking what was your experience like at school is not very measurable. This is a complex question that can have many different answers, and so it’s better answered through qualitative research. Another consideration is if your indicators are achievable.

For example, can you collect the data needed for your indicator with the resources and budget available? You also want to consider if your indicator is relevant. Is your indicator a valid way to measure the outcome of interest in the studies population and setting? Importantly, is it valid for capturing the experiences of people with disabilities? You should consult with experts and people with disabilities familiar with the study setting before you decide if your indicators are relevant. Finally, indicators should consider the time frame of data collection. Some indicators may change at different points of the year due to holidays, politics, or seasons. These changes are particularly important to consider when you’re tracking changes over time.

For example, in some settings children are more likely to miss school during the planting and harvesting season. If you want to measure changes in school attendance before and after new education policies were put in place, you’d want to compare data collected at similar times of the year, so that you don’t over- or underestimate the impact the policy. Studies that track changes over time should also consider if indicators are likely to change over the study time period. For example, you might not expect to see much change in primary school completion six months after the implementation of a new education policy, but indicators such as dates of school missed or performance on tests might change during this time.

Let’s now go through some examples so we can explore how to select indicators that are appropriate to your research question and for people with disabilities. An example. Let’s look at access to improved sanitation. Let’s say we have a study that wants to answer the research question how does access to improved sanitation differ between people with and without disabilities? We will use as our indicator the proportion of people living in households with access to an improved sanitation source. Data collectors observe each household’s main sanitation facility during a household visit and then classify the type of facility as improved or unimproved based on international guidelines, such as those used in the Sustainable Development Goals. Is this indicator and underlying data SMART?

Is it appropriate for capturing the experiences of people with disabilities? Let’s go through the SMART criteria. Is the indicator specific? The indicator is somewhat specific. Having an improved sanitation facility is an important indicator of having improved sanitation. However, by asking the question at the household level, it assumes that all household members use the same facility and in the same way. People with disabilities may not use the same sanitation facility as other household members. And even if they do, they may face other barriers that affect their access. Is it measurable? Yes. Data collectors can observe the type of sanitation facility, and, trained properly, can classify it as improved or unimproved based on the guidance provided. Is it achievable?

We will need to budget time for training data collectors to recognise and classify different sanitation facilities. We also need to budget time for data collectors to make the observation of the facility at each household they visit, which may take some time if the facility is not within the household. Is it relevant? This is a valid measurement that is based on internationally agreed-upon standards. However, it might not be the most relevant measure for capturing access to sanitation amongst people with disabilities, as it doesn’t consider access challenges such as if the facility can be used identically or independently by people with different impairments.

Measuring access at the individual level, including questions about access issues faced by people with disabilities, could make this indicator more relevant. Is it timely? In this study, we only asked about the main sanitation facility. But people may use different facilities throughout the year. For example, some sanitation facilities might be unusable during rainy seasons. People with disabilities may be particularly affected by seasonal disruptions. For example, people with mobility limitations may find it more difficult to travel to a different sanitation facility. We could take into account seasonal changes in access by asking about all facilities used throughout the year, or, if tracking changes over time, repeating data collection at a similar time of year.

Overall, do you think this is a good indicator for our research question? How could this indicator be improved? Let’s see the impact of different approaches to measuring access to improved sanitation amongst people with disabilities. The first is the same indicator we just discussed. It compares the number of people living in a household with that improved sanitation facility. When we use this indicator, we saw virtually no difference between people with and without disabilities. 95% of people with disabilities had an improved facility versus 96% for people without disabilities.

However, when we used a different indicator which looked at individual access, such as if a person could use a facility without coming into contact with faeces or independently when needed, we saw a very significant differences between people with or without disabilities. With this measure, 88% of people without disabilities had no access challenges. However, only 60% of people with disabilities face no access challenges. This changes the answer to our research question and what we might recommend for policy and programmes, as now we can see a big difference in sanitation access between people with or without disabilities. The previous example, it’s important to always consider if indicators capture the experience of people with disabilities.

This applies to many types of indicators you might want to use in studies covering a range of different topics. For example, if you’re looking at education, you should consider whether education is inclusive. Are children with different types of impairments supported in the learning process? Similarly, if you’re looking at access to help, you may want to consider barriers that are particularly important for people with disabilities, such as physical and informational accessibility. You may also want to consider access to disability-specific health care, such as assistive devices or rehabilitation. Or if you’re looking at violence and social participation, it’s important to consider discrimination and stigma due to disability.

In summary, you should now understand how to design indicators that are appropriate for your research questions, and evaluate if indicators adequately capture the experiences of people with disabilities.