How do we improve the implementation of an intervention?

In this step, we will learn about process evaluations. We will start by defining what a process evaluation is, then go on to talk about what are the benefits of process evaluations and the key domains that are studied within process evaluations. The information in this step is drawn from the Medical Research Council guidance. This step will be a brief overview. And if you want to read more about this type of research, there’s a link to the full guidance in the see also section.

In the last step, you heard about impact evaluations and how they are used to understand whether an intervention is effective or not. According to the MRC definition, a process evaluation is a study that aims to understand how an intervention works. It examines three key areas or domains– implementation of an intervention, so how it was delivered to the population, mechanisms of impact, so how the intervention worked or not, and contextual factors where it was delivered. These three components together help to understand how the intervention worked, why it did not work, and whether it could be suitable for delivery in a different setting.

A process evaluation complements an impact evaluation by providing more than just the outcome.

Process evaluations are primary studies in that they collect new data. They aim to evaluate a specific intervention and take place alongside a larger experimental study, such as alongside an impact evaluation. Mixed methods are usually used, which, as you heard earlier in the course, involves both quantitative and qualitative methods of data collection.

Using the example from the previous presentation, an impact evaluation can show us that the intervention reduced anxiety by 20%.

A process evaluation asks, what was it about the delivery of this programme that made it work, how did it work, and why did it work in this particular setting. Conversely, process evaluations also help us to understand why interventions did not work. Process evaluations examine these three domains of implementation, mechanisms, and contextual factors. So an impact evaluation tells us whether an intervention is effective, but doesn’t tell us exactly how the intervention works. So an impact evaluation tells us whether an intervention is effective, but doesn’t tell us exactly how the intervention works. For example, impact evaluations would not be able to tell us whether the intervention would still be effective in another context.

Now we will examine the three domains of process evaluation in more detail. When we look at the implementation domain, we are interested in what was delivered in practise. There are several key areas that can be assessed here. For example, we might be interested in the quality of implementation. Was the intervention or programme delivered as it was planned or were adaptations made? We might also be interested in the dose. Did all programme participants or recipients receive the same quantity of the intervention? Or did some people receive more or less? We can also assess reach, which is the proportion of the intended audience reached by the programme and how representative they are of that target population.

Finally, barriers and facilitators to implementation can be assessed. The mechanisms of impact domain is trying to assess how the intervention worked or did not work. And if the intervention was effective, were there any unintended or negative consequences? For example, the programme earlier may have reduced anxiety in the target population. But was it too time-consuming or costly for the participants for long-term change? Finally, we can examine how the target audience interacted with the intervention. Was it acceptable and relevant to them? Lastly, contextual factors are factors that might be external to the intervention. This might be people, structures, or culture, politics, or religion. These factors can have an impact on delivery implementation and mechanisms of impact.

When conducting process evaluation, we need to consider possible external factors that might influence this. If we look at an example, an intervention delivered by clinical officers in a hospital in Malawi might perform differently in a country in Southeast Asia which has a different health system and different cultures of health workers.

In summary, a process evaluation is a study that aims to understand the functioning of an intervention. So not just whether an intervention works, but how it works. These studies often complement an impact evaluation. The key areas which a process evaluation examines are implementation of an intervention, mechanisms of impact, and contextual factors. In the next step, you will hear about an impact and process evaluation conducted in the Maldives to assess the disability allowance.