How to approach your framework

An effective evaluation framework should reflect your priorities in terms of learning, stakeholders and applications. Achieving those priorities relies on getting the “when” right, gaining a good understanding of the “why”, and planning to involve the right “who”.

The When

Design of an evaluation should take place at the start of a project, as part of the overall planning. Evaluations bolted on in the latter stages often lack critical data and their formative value is undermined. Remember too that an evaluation framework is a live concept and document that you will need to return to on an iterative basis as more information becomes available and plans evolve. You may want to use it to help stakeholders to keep on track and to support moments for reflection during the plan (see Week 2, Activity 2: Creating your evaluation plan).

Step 1: The Why

Being clear about the purpose of an evaluation is important. “How will your evaluation be used?” is a critical design question. An evaluation can be either:

• Summative: a cumulative evaluation which draws conclusions at the end of a programme, often measuring overall change from a baseline or contextualised by other standard benchmarks.
• Formative: iterative, stepped evaluation and testing which feeds into the ongoing development and adaptation of a programme at regular intervals. Research methods and data capture are likely to adapt over time as well.
• A hybrid of the two: as is often the case in real life situations. 

Kick off the process by asking: 

• What change are we facilitating? 
• What difference will the full story of this project make? 
• What specific needs must this evaluation serve?

Step 2: The Who

Although it can feel more time-consuming and complicated to involve lots of people in development of the framework, it is likely to yield better results in the long-term, creating a well-rounded understanding of hopes and ambitions from multiple perspectives. Stakeholders can also help to flush out implicit hopes and fears and to fill gaps in the project logic and the plan. They will also help you sense-check the brief. Most importantly, perhaps, giving people a stake in the design of the evaluation is also likely to encourage their interest and support during the project and heighten their receptiveness to its conclusions and learning at the end.

Step 2 is to ask: Who are your stakeholders? As an evaluator, you need to think about all the people involved in a project:

• People involved in delivery (e.g. artists and volunteers)
• People benefiting from the programme (e.g. audiences or local communities)
• People who will benefit from the evaluation (e.g. yourselves and your funders)

You need to consider who needs to be counted, whose voices need to be heard, whether or not some voices need to be prioritised, at which point and in what way, especially if resources are limited. It can be a challenge to balance the Principles of being people-centred and inclusive with being proportionate. 

Once you have identified all project stakeholders, the next questions are: who should be involved in the evaluation, and how? There are a few right and wrong answers and these might depend on the resources you have and the priorities you have established. Thinking about the Principles can also help guide you. 

If you are planning on workshopping this process, there are lots of facilitation tools to help manage “divergent thinking” – getting lots of ideas – with “convergent thinking” – agreeing together which are the most important.

In the next step, you will learn how thinking through the logic of the project will help you define the key research questions that the evaluation is trying to answer.