Comparative judgement

This case study appears as part of our report entitled, ‘Testing the Water’. Permission has kindly been granted by LKMCo and Daisy Christodoulou to include the case study as part of this course. You can read the report in full with other case studies here

Improving the reliability of teachers’ judgements – No More Marking and comparative judgement

No More Marking is an organisation developing comparative judgement. Comparative judgement is based on the principle that people are better at making comparisons between pieces of work than at making absolute judgements about quality. No More Marking has developed software that enables teachers to compare two pupils’ work (for example, two essays exploring the question: ‘What were the causes of the Second World War?’). Each teacher simply judges which response he or she thinks best addresses the question. Once enough teachers have judged enough pieces of work, it is possible to see where an individual pupil’s work fits into the broader distribution of achievement, and standardise pupils’ achievements to see how they compare with other pupils, nationally.

The government said it is going to explore the potential for comparative judgement in supporting assessment in primary schools in its assessment consultation response, published in September 2017.

Daisy Christodoulou is No More Marking’s Director of Education, and said she feels the approach has three key benefits:

1. Reliability: Pilots of the comparative judgement approach suggest it is more reliable than methods traditionally used to assess pupils’ written work.

2. Validity: The assessment of written work using marking criteria has historically proven problematic, leading to pupils jumping through hoops (for example, using clunky or nonsensical frontal adverbial phrases). Comparative judgement involves making a holistic evaluation of a piece of work. In Christodoulou’s words, “you’re actually assessing the thing you want to assess.”

3. Efficiency: Comparative judgement enables markers to assess pupils’ written work more quickly. Christodoulou explained that the average time it takes users of comparative judgement to make a judgement is 38 seconds, far quicker than the time taken to grade an essay using marking criteria.

Image taken from the No More Marking website of pupils’ work, as displayed to markers on screen:

Christodoulou also explained that, while in its relative infancy, comparative judgement could be used for peer assessment by pupils too. The approach enables evaluation of the sorts of features judges identify and reward when assessing work, “so if you set up pupils as judges you can see to what extent they’re agreeing with the teachers.”

While the evidence supporting comparative judgement is encouraging, the approach represents an entirely different way of evaluating pupils’ work: “If you’re used to marking with criteria and having those moderation meetings it’s really different and it can feel very subjective. The weird thing is it’s much more rigorous than traditional marking but it doesn’t feel it.” Other challenges are practical, such as inputting pupils’ work before conducting the assessment.

If you choose to focus on this case study as part of this week’s learning, you can share any initial reflections and questions with the course community in the comments space below.

• How might any of these approaches be applied in your own context to solve a challenge you’ve identified?

When you are ready, click the ‘Mark as complete’ button below and then select ‘Discuss your learning’ to connect with other course participants.