Assessing impact in practice – what does it look like?

Now that we understand the importance of measuring impact, we need to think about how to actually do it. Although we’re going to be look at some of the techniques that researchers use for measuring impact, you don’t need to have any experience with statistics. We’re going to give you a broad overview of what good study design and evaluation looks like.

In theory, we know what we need to do: find a control group that has similar characteristics to the treatment group, both observable and unobservable. In practice, however, finding an appropriate counterfactual can be challenging.

As discussed earlier, the most straightforward way of finding an appropriate control group is to randomly assign treatment. However, sometimes this may not be possible.

• In some cases, an organisation has already chosen and committed to implementing a policy in a certain way.
• In other cases, we want to evaluate the impact of a policy that was implemented in the past.

In all of these cases, it is still important to try and evaluate impact so we shouldn’t give up just because the statistics part will be harder!

A number of methodologies exist that can be used to identify the causal impact of a policy or intervention.

Difference-in-difference

One of these is called difference-in-differences. Using this methodology, we compare the difference in outcomes before and after a specific event (e.g. policy change) between a group that has been affected by the event and a group that has not been affected. The important assumption that we make when doing this, is that any changes that happened to the group not affected by the event during this time, would have influenced the affected group in the same way.

Any remaining difference can be attributed to the event itself. It is important to remember that this assumption is quite strong but if we have data for multiple time periods prior to the event then we can test to see if it holds. For example:

• Suppose we want to estimate the impact of a microfinance programme on income, and that the programme specifically targeted the poorest villages.
• Comparing incomes between villages that received the programme, and those that did not, would give a misleading estimate of the impact.
• This is because non-programme villages were richer than programme villages prior to the intervention.
• But if we think that this difference in the level of income would remain the same over time in the absence of the programme and we have data from this point in time then…
• Any difference between the post-programme difference in income and the pre-programme difference could be attributed to the effect of the microfinance programme.

Regression discontinuity design

Another methodology that can be used is called regression discontinuity design. This methodology exploits a discontinuity around a threshold above or below which an intervention is assigned. By comparing observations lying closely on either side of the threshold, it is possible to estimate the effect of the programme.

Suppose a government offers an income support programme for any individual earning less than $100 per month. We would expect individuals who earn $99 to be very similar to those who earn $101 and therefore a comparison of outcomes between these groups can be used to estimate the impact of the programme.

While this approach can be very effective, it is only possible to use it when there is an identifiable threshold with a large number of individuals very close to both sides.

Instrumental variables

The final methodology that we will discuss is called instrumental variables. Suppose we would like to estimate the impact that mobile phone use has on political mobilisation in Africa. The difficulty is that there may be a number of other variables that affect both mobile phone use and political mobilisation (e.g. income, social connections). This makes isolating the direct impact of mobile phones challenging.

We can overcome this problem if we can find another variable that is related to our variable of interest (mobile phones) but is not directly related to our outcome variable (political mobilisation). This is called an instrumental variable.

Some authors (Manacorda and Tesei) have used incidence of lightning as an instrument to estimate this relationship as it makes the mobile phone connection worse but should not have a direct effect on political mobilisation. This approach can be very effective if you can find a good instrument but in many cases this can be very challenging!

No one methodology is perfect. Different situations will call for different methodologies and the key is to find the best one to answer the particular question at hand!

• Think about some personal characteristics that have influenced whether or not you experienced a particular event (e.g. buying a house, going to university, starting a business).

• Do you think that these characteristics might influence other outcomes in your life that would also be influenced by that event?

• Why does that make it so difficult to understand the true impact of the event?

For example, suppose you are very ambitious and hard-working and decide to go to university. Could you attribute all future success in your career to having earned a university degree or might you have been successful anyway because of those personal characteristics?

Tara Mitchell is an Assistant Professor at the Department of Economics, Trinity College Dublin and a consultant to the World Bank. Her research focuses on the microeconomics of development with a particular interest in agricultural markets and informational problems.