Research example: impact and process evaluation in the Maldives

This week, you’ve learned about impact evaluations and process evaluations. Let’s look at our study in the Maldives in more detail. Here, we used mixed methods to evaluate a social protection intervention designed to support people with disabilities. In the Maldives, people with disabilities can receive a monthly cash benefit called the disability allowance. It is a fixed amount of about $130 and it has not changed since the intervention was introduced in 2010. In order to see what changes, if any, were needed to improve the intervention, the Maldives National Social Protection Agency commissioned an impact evaluation. This was to detect, if, indeed, that the allowance was having an impact on the lives of people with disabilities.

To explore this, we conducted a process evaluation and then an impact evaluation. As you’ve heard in the last step, a process evaluation is a study designed to understand how an intervention functions. This looks at how the intervention is delivered, by whom, at what cost. It also looks at contextual factors that may be affecting the delivery and uptake of the intervention. In the Maldives study, we conducted a process evaluation to understand the disability allowance programme by collecting data through document reviews and in-depth interviews with implementers, stakeholders, and beneficiaries.

In collaboration with programme implementers, we also developed a theory of change, which is a map of how the intervention is expected to have an impact on beneficiaries– in this case, people with disabilities. Here, you can see a portion of the theory of change we developed for this study. The boxes in red indicate the intended outcomes of the programme– reduced poverty, improved health, participation, and quality of life. The other boxes indicate other processes that are expected to lead to this. We then used this information to design the impact evaluation. We used quantitative and qualitative methods in parallel, to explore whether the components depicted here worked as anticipated to produce an impact.

You can read more about the methods we used in the lesson on mixed methods in week 1. We knew from the process evaluation that the disability allowance was designed to improve beneficiaries’ functioning and participation by giving them access to assistive devices. However, the impact evaluation data showed that this did not always happen. That is, allowance beneficiaries were not always referred to the programme that provided assistive devices. This meant that the intervention did not improve beneficiaries’ participation as intended. We also knew from the process evaluation that the intervention was expected to improve people’s access to services.

The impact evaluation showed that, because disability-related services were scarce in the country, people were spending vast amounts on travel abroad or to the city, and the allowance was not enough to cover it. Lastly, the disability allowance was expected to protect beneficiaries from poverty by providing them with the monthly cash benefit with no conditions on how it was spent. However, the impact evaluation revealed that the monetary value of the allowance was insufficient to push people out of poverty and cover the high costs related to disability. Based on these findings, we were able to develop recommendations about how to improve the effectiveness of the intervention. We disseminated these findings in a number of ways.

We held in-depth discussions with the implementers to explain why their intervention had low impact. Because we had developed the theory of change with them, it was easy to communicate to them which processes need to be strengthened in order to improve the impact of the intervention. We conducted a series of small seminars throughout the country. We found that this was more effective in reaching various different audiences than having one major seminar in the city. We were able to reach study participants, community members, local government, DPOs, government agencies, international NGOs, academia, and policymakers. We also developed communication material like infographs, making versions in English and in the local language. This ensured that the important figures could be disseminated widely and easily.

In summary, with this case study, you should now be able to recognise how a social protection intervention can be evaluated, understand how a process evaluation can complement an impact evaluation, and appreciate a few ways of disseminating findings.