Assessing impact in practice – what does it look like?
- In some cases, an organisation has already chosen and committed to implementing a policy in a certain way.
- In other cases, we want to evaluate the impact of a policy that was implemented in the past.
Difference-in-differenceOne of these is called difference-in-differences. Using this methodology, we compare the difference in outcomes before and after a specific event (e.g. policy change) between a group that has been affected by the event and a group that has not been affected. The important assumption that we make when doing this, is that any changes that happened to the group not affected by the event during this time, would have influenced the affected group in the same way.Any remaining difference can be attributed to the event itself. It is important to remember that this assumption is quite strong but if we have data for multiple time periods prior to the event then we can test to see if it holds. For example:
- Suppose we want to estimate the impact of a microfinance programme on income, and that the programme specifically targeted the poorest villages.
- Comparing incomes between villages that received the programme, and those that did not, would give a misleading estimate of the impact.
- This is because non-programme villages were richer than programme villages prior to the intervention.
- But if we think that this difference in the level of income would remain the same over time in the absence of the programme and we have data from this point in time then…
- Any difference between the post-programme difference in income and the pre-programme difference could be attributed to the effect of the microfinance programme.
Regression discontinuity design
Want to keep
Trinity College Dublin online course,
Achieving Sustainable Development
While this approach can be very effective, it is only possible to use it when there is an identifiable threshold with a large number of individuals very close to both sides.Suppose a government offers an income support programme for any individual earning less than $100 per month. We would expect individuals who earn $99 to be very similar to those who earn $101 and therefore a comparison of outcomes between these groups can be used to estimate the impact of the programme.
Instrumental variablesThe final methodology that we will discuss is called instrumental variables. Suppose we would like to estimate the impact that mobile phone use has on political mobilisation in Africa. The difficulty is that there may be a number of other variables that affect both mobile phone use and political mobilisation (e.g. income, social connections). This makes isolating the direct impact of mobile phones challenging.We can overcome this problem if we can find another variable that is related to our variable of interest (mobile phones) but is not directly related to our outcome variable (political mobilisation). This is called an instrumental variable.Some authors (Manacorda and Tesei) have used incidence of lightning as an instrument to estimate this relationship as it makes the mobile phone connection worse but should not have a direct effect on political mobilisation. This approach can be very effective if you can find a good instrument but in many cases this can be very challenging!No one methodology is perfect. Different situations will call for different methodologies and the key is to find the best one to answer the particular question at hand!
- Think about some personal characteristics that have influenced whether or not you experienced a particular event (e.g. buying a house, going to university, starting a business).
- Do you think that these characteristics might influence other outcomes in your life that would also be influenced by that event?
- Why does that make it so difficult to understand the true impact of the event?
Tara Mitchell is an Assistant Professor at the Department of Economics, Trinity College Dublin and a consultant to the World Bank. Her research focuses on the microeconomics of development with a particular interest in agricultural markets and informational problems.
Achieving Sustainable Development
Our purpose is to transform access to education.
We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.
We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.