Impact Measurement Framework

We have seen before that transparency, causality, accountability, and acceptance are drivers for the use of impact measurement tools. In this way, we answer the question ‘Why impact measurement?’. Another question is ‘What is the benefit?’. Here, we differentiate between efficacy, effectiveness, and legitimacy. All three are good reasons to conduct an impact measurement tool. But a clear benefit for the organization will only occur if you choose the right method for your purpose.

In the following, we differentiate the benefits of efficiency, effectiveness, and legitimacy. As you can see in the figure below, the combination of the specific reason and benefit leads to different impact measurement tools. Thus, before selecting a tool, you should always check what necessities you have to comply with.

The simplest measurement is cost-efficiency. But for many situations, this will be perfectly right. For instance, a well-established childcare service will not need to measure its impact every year, but it will need a clear statement on the resource allocation and cost sensitivity.

^{Order of impact measurement tools (Center for Philanthropy Studies, 2018)}

Measurement of effectiveness can address different aspects, depending on the grade of quantification and monetization. Cost-effectiveness analysis helps to understand for one single impact dimension how many resources are needed to reach a given goal (ie X$ for one day per person without hunger). The cost-utility analysis uses standardized utility indicators to compare the costs of different interventions. Typical utility indicators are quality-adjusted life years (QALY) or disability/disease-adjusted life years (DALY). A typical result of a cost-utility analysis is intervention A causes X$ per QALY. Finally, the cost-benefit analysis combines the results of one or more impact dimensions of one intervention (ie the intervention caused a total impact worth X$).

Finally, we look at methods that help strengthen legitimacy. The Social Reporting Standard will be introduced in the following video. By the way of reporting about your activities, you may improve your legitimacy.

Program Outcome Model

The program outcome model (POM) is today’s basic model of many other impact measurement tools. The POM introduced the value chain approach to impact measurement and is very well documented. Thus, it is perfect to highlight cause-effect-relations between input, activities, output, and further outcomes. Following this link, you will find a step-by-step-explanation of the model.

Social Return on Investment

The social return on investment (SROI) has gained a lot of attention in both research and practice. Basically, it follows the concept of the program outcome model. The difference is that all outcomes and impacts are monetized and thus, the final result is not only a cause-effect model but enables to make statements about the benefit of intervention for different stakeholder groups. A general formula used to calculate SROI is the following:

[SROI = frac{social impact value – initial investment amount}{initial investment amount} * 100]

Although the SROI provides a clear result, it is often criticized as inaccurate, because it requires many assumptions in order to monetize all impact indicators. In this process, many over-or underestimations can occur.

Randomized Control Trials

Finally, the gold standard of legitimizing your work is randomized control trials (RCTs). Often applied in development aid projects or clinical studies, the RCT offers a comparison between an intervention group that benefits from a specific project and a control group that receives no (or limited) support. If both groups are selected randomly out of a full basic population, one can receive clear evidence about the impact of the intervention (Pomeranz, 2017). Because of this scientific and objective approach, results out of RCTs obtain high acceptance. However, RCTs are expensive and time-consuming and, thus, only suitable for large and highly innovative projects.

Paper & Pencil Task

Think about your organization, your projects, and the relevant stakeholders. What kind of impact measurement tool is suitable for you? The following video will help you to better understand different stakeholder groups and the relation to impact measurement.

References

Pomeranz, D. (2017). Impact Evaluation in Public Economics: A Brief Introduction to Randomized Evaluations and Comparison with Other Methods. In Public Finance Review, 45(1), 10-43.