Skip to 0 minutes and 3 secondsANDREA MAPPLEBECK: Thank you, everybody, and welcome. Here we are for another question and answer session with our experts, Dylan and Chris. This time, we're doing it across three different continents, which is an exciting experience. So we shall see how that goes. So thank you, Chris and Dylan, for your time. Thank you too to the participants on the course on this Planning for Learning online course, for your questions. Without further ado, we shall ask Dylan and Chris for their responses and get their expert ideas to help us all develop our practise.

Skip to 0 minutes and 35 secondsSo the first question, actually, is a point that's been raised by several people across the course who-- because one of the things we've talked about on Planning for Learning is actually this idea about decision-driven data collection. And some people would really value further clarity on this. So I thought, as it came under a step where Dylan was talking, we'd get Dylan to actually respond to that one first. So, Dylan, if you could help our understanding, that would be really good. Thank you.

Skip to 1 minute and 3 secondsDYLAN WILLIAM: Yeah, thanks, Andrea. I think it's best illustrated by a teacher I saw a while back. She was teaching a lesson on mass and weight. And five minutes before the end of the lesson, she asked the students to explain-- on a little three-by-five index card-- the difference between mass and weight, a technique that many teachers use called an exit ticket. So at the end of the lesson, when the bell went, the teacher said to the students to leave and to hand in their exit tickets as they left. And then the teacher read through the exit tickets. And then she put them in the bin, which surprised me. And I asked her, why did you put them in the bin?

Skip to 1 minute and 40 secondsShe said, "Well, I know where to start tomorrow's lesson." I said, what did you learn? She said, "They all got it right. I'm moving on." I said, what would you have done if they'd all got it wrong? She said, "I'd have taught it again but slower and louder." She said it as a joke. She said, "I'd have taught it again in a different way." I said, what would you have done if half the students had answered correctly, and half the students had answered incorrectly? She said, "I would have kept two of the cards, one with a good answer, one with a not-so-good answer. And then I would begin tomorrow's lesson with these two cards on the visualizer."-- the document camera.

Skip to 2 minutes and 13 seconds"And then I'd ask the students to vote for one or the other and take the discussion from there." I said, why didn't you give the student-- the individual students feedback? She said, "I couldn't. They didn't write their names on the cards." I said, why didn't you get them to write their names on the cards. She said, "That would be really stupid. If I wanted to give individual feedback, I'd have gotten them to write their answers in their science notebooks, which already have their names written on them." And that's when I realised that what this teacher was doing is so smart. She wasn't doing data-driven decision making. She was doing decision-driven data collection.

Skip to 2 minutes and 44 secondsShe just wanted to decide where to start tomorrow's lesson. And she wanted to collect the least burdensome amount of information that would help her make that decision in a smarter way. She did not want to spend 30 minutes giving students individual feedback on those 30 exit tickets. She wanted a quick read on the whole class's understanding so that she could make a good decision for the learning needs of the whole class. In other words, she started with the decision and then figured out what data, what evidence, will help me make that decision in a smarter way. And I think it's important to clarify these two terms, data and evidence, because data is anything. Its numbers, its words.

Skip to 3 minutes and 24 secondsIt's anything that people say. It becomes evidence when it's used in support of a claim. So when you say that a certain piece of data has a particular meaning, then data becomes evidence. And so the important thing is it's just data until you decide it has a meaning. And then it's evidence. But the important point is, start with the decisions, then you'll always know what to do with the evidence that you get. If you haven't got any clear idea, then it's just data. It's just numbers sitting there waiting for somebody to give them some meaning by asking a particular question.

Skip to 3 minutes and 58 secondsAnd that's why we see so many schools drowning in data but with very little information because they collected the data without any clear purpose in mind.

Skip to 4 minutes and 6 secondsANDREA MAPPLEBECK: Chris, have you got anything to add to that?

Skip to 4 minutes and 10 secondsCHRIS HARRISON: My example is from the classroom as well. Mine is-- rather than collecting evidence at the end of the lesson or data to use as evidence at the end of the lesson, it's actually collected during the lesson. So I'm thinking of a class I went to see of 17-year-olds who were studying geography. And the teacher, Heidi, had them doing a peer-assessment exercise because they'd come almost to the end of a project they were doing. And so she put them into groups of three. And they were taking each other's folder and going through it. And two of them were giving guidance to the other student.

Skip to 4 minutes and 48 secondsWhat Heidi was doing was going around with a piece of paper, listening in to what was being said. Rarely did she intervene with what was going on. It was more that she was actually picking up information from what the students were giving to one another to then decide at the end of the lesson what she was going to do in terms of summing up the lesson. So she was actually-- just as Dylan's explained-- collecting evidence of what was going on in order to use it at the end of the lesson to draw everybody back onto that pathway that she wanted for learning.

Skip to 5 minutes and 20 secondsSo these sorts of activities happen when teachers are planning how to get assessment to run alongside the learning so the assessment really does inform the learning. And that's why it's a much better way, taking it from the decision-making than to actually just collect the data, as some teachers do. It's not for accountability. It's for actually informing the learning.

Skip to 5 minutes and 45 secondsDYLAN WILLIAM: And I think it's important to note that this is often a feature of teaching in Japanese classrooms, for example. In American classrooms, in British classrooms, teachers often try to help individual students. In Japanese classrooms, teachers will walk up and down the aisles, looking at what the students are doing. But they'll be planning what to do with the whole class. And I know that many schools in England are now experimenting and exploring this idea of whole-class feedback, which is far more effective. And it doesn't get everybody.

Skip to 6 minutes and 14 secondsBut the important point is, it's better to try to put the class's learning in one go for the whole class before you resort to individualised, one-to-one help because often you can put a lot of kids back on the right track in one go rather than having to individualise everything.

Skip to 6 minutes and 32 secondsANDREA MAPPLEBECK: Thank you. That's really interesting, really interesting. That resonates with some stuff I found out in my research. So thank you both very much for those ideas. And already thoughts are coming into my mind for when I sum up at the end. I think that's provided great clarity from you both. So thank you very much because I know that will help lots of our learners on the MOOC. So moving on to question 2, which is a question raised from Laurence. So, Chris, I'm going ask this one to you first. Laurence has asked what you would consider good practise in terms of keeping evidence from formative assessments?

Skip to 7 minutes and 4 secondsCHRIS HARRISON: I think you need to consider what you're collecting that evidence for. So, in most cases, you don't because learning moves on. And once it's moved on, then you're at a different place in terms of the children's learning. So the collecting data on individual students, it's not that useful, although you might want to look back and sum up if-- most teachers actually manage to do that quite well [INAUDIBLE]..

Skip to 7 minutes and 35 secondsWhat we found on our last project-- we were looking at a classroom assessment with inquiry learning in STEM subjects-- was that the teachers actually selected particular activities where they thought it was worth writing something down because they felt that they'd actually been doing inquiry and investigations over a couple of weeks. And they just wanted to see-- to use the data they were collecting, both for summative purposes and performative purposes. And so in those sorts of lessons, they would write down-- it used to be per group rather than per individual. It was just really sort of keeping the track on where the classes were going in terms of, could they now raise testable questions?

Skip to 8 minutes and 17 secondsOr were they now able to see how much data they needed to collect to try and answer the data and so on? So, again, it was more collecting evidence to inform future teaching and future learning than actually doing it in terms of reporting. However, some of them did use some of the evidence they collected when they actually had to write reports for those students to their parents at the end of the year because they found looking back over the few comments they'd written to each of those key activities that they'd focused on enabled them to talk about things like student's aptitude for teamwork or the competence in terms of communication, et cetera, as well as commenting on the more science-type criteria, such as, can they raise a hypothesis?

Skip to 9 minutes and 6 secondsOr can they draw a graph or whatever? So sometimes if you do need to do some reporting or you do need things in order for more summative purposes, then it might be worth noting things down and think about what might be useful for that. But most of the time, I would say, it's not really worth writing down when you actually do an assessment for learning because things move on so quickly. A few schools, particularly in the UK-- or, well, particularly in England rather than the whole of the UK-- the schools like to check that the teachers are actually giving feedback.

Skip to 9 minutes and 42 secondsAnd so they have things called book trolls, where a senior teacher will look through just to check the teacher is giving feedback. Again, that can be done in a simple way that doesn't take much time because its purpose really is just checking on teachers rather than checking on students. And, again, I wouldn't spend a lot of time on that. But if you have to do it, then do the minimum that you can.

Skip to 10 minutes and 8 secondsDYLAN WILLIAM: I think this question comes from an assumption that teachers have to prove they've been doing some formative assessment. And I think that's-- I just think that this is bureaucracy gone mad. And managers are often looking for these kinds of things. And I just think we have to grasp this nettle, and just say it's a bad idea. If the reason you're recording your formative assessment results is to prove to somebody else you've been doing formative assessment, then just stop. The best formative assessment often leaves no written evidence. The evidence is in a smarter decision by the teacher about what to do next. And so I think the crucial thing is why are you doing it?

Skip to 10 minutes and 49 secondsYes, you can record what you see students doing. But the danger, of course, is that you might be measuring performance rather than learning. So you see students doing something in a particular task. And they can do it today. And they could do it tomorrow. But they can't do it in six weeks' time. So I think we have to be very careful about assuming that because we've seen students doing something once in a classroom that they've actually learned it. And so I come back to this idea, why are you doing this? What decision is going to help you make better? In England, some teachers were giving verbal feedback.

Skip to 11 minutes and 22 secondsAnd because there wasn't any evidence about this, some schools got a little rubber stamp that said, "verbal feedback given," as if that was a way of proving you've been doing it. What I don't understand is why anybody is convinced by that. Because if the teacher is not giving verbal feedback, what's to stop them saying they've given it when they haven't? It's just bureaucracy taking over the process. And I think we have to put the learning front and centre. We'll get good formative assessment when student learning is at the forefront, not proving you've been doing particular kinds of processes for administrators to look and see whether you're doing the job properly.

Skip to 11 minutes and 55 secondsAnd I think that we need very clear messages on that.

Skip to 11 minutes and 59 secondsANDREA MAPPLEBECK: I'm sure there's lots of teachers who will be listening to what you've both said and will be cheering after hearing that. So that's lovely. Thank you. And moving on now to our next question, which is from Jamal. And this is now looking at the place of summative assessment. So within the philosophy of formative assessment, how does summative assessment fit? So, Dylan, your thoughts on this. Thank you.

Skip to 12 minutes and 24 secondsDYLAN WILLIAM: Well I think the first stage-- the first step, if you like, is to be clear about what we mean by formative and summative. Formative and summative aren't descriptions of assessments. They're descriptions of the uses to which the information is put and particularly the conclusions that are drawn. So, for me, formative and summative are descriptions of the kinds of conclusions we draw on the basis of assessment evidence. So if I conclude that this child knows his number facts, then that's a summative conclusion. If I see from the same evidence that he's having trouble with the seven times table, that's a formative conclusion.

Skip to 12 minutes and 59 secondsSo the same test, the same assessment, even the same assessment evidence, can be used both summatively and formatively. So I think what we need is productive synergies between the two. And so, for me, because learning is a change in long-term capability, then we do need periodically to review whether students have retained what they've been taught. And so we do need assessments that are designed primarily just to support evidence of mastery. But even that can be used formatively. Because if a student hasn't shown evidence of mastery, then you do something about it by actually intervening to actually help that student reach mastery. So, for me, every assessment is potentially both formative and summative.

Skip to 13 minutes and 44 secondsBasically, even at university, the examinations you do at the end of your undergraduate studies is summative. You get the degree. But it's also formative in that it can actually then focus what kinds of postgraduate activity you might be doing. So, for me, it's about thinking about when is the right time to do this. And remember also that, as Alfie Kohn points out, grading, judging, usually stops the learning for at least a little while. And so, for me, the problem I see in most countries when I look at what's going on is we tend to tell students where they are too often and don't tell them how to get better.

Skip to 14 minutes and 23 secondsBecause what the research seems to show-- I mean, it's very complex to read at all-- but the research that I've read suggests that it's very hard to tell students where they are and how to get better at the same time. So I think, for teachers, the advice I'd be giving is be clear. Why are you doing this? Are doing it in order to figure out what to do next and how to get better? Or are you just collecting evidence for a particular summary of the students' achievement at a particular point in time? And as long as you're clear about that, then I think that's-- then the decisions you need to make will pretty quickly fall into place.

Skip to 15 minutes and 4 secondsANDREA MAPPLEBECK: And Ann would you like to add to that?

Skip to 15 minutes and 8 secondsCHRIS HARRISON: So, for me, I think the way that most teachers we work with tend to keep formative in the foreground and summative in the background. And it might be the summative is-- works out in advance of what do I need summative evidence for? So when will there be times when I do really need to have some summative data in order to do some particular tasks that I've got to do, either sorting groups or writing reports or whatever? But it's the formative that's important, the use of the formative that's important, and keeping it in the foreground.

Skip to 15 minutes and 43 secondsAnd I think when you do that, and when you get teachers to, when they're working with students in the classroom, to use the formative data with the language that goes with it. So, for example, I don't let my pre-service teachers use the term, "marking." I tell them, please don't mark children's books. I want you to say to the kids, "Can I have your books in because I want to give you some feedback." So they know the stuff they're getting back is about guidance and it's formative. And it might be at various times that they need to give them a test or they need to give them an activity where they need to give them a grade.

Skip to 16 minutes and 19 secondsAnd they're just honest and open about that at the beginning. And they might say, this is going to help-- this particular activity is going to help me see where you've got to. And it will mean that we can then plan the next bit. So just as Dylan said, it's initially for summative. But there's a formative tag on that that takes them forward. So we're always looking at formative and summative working together to actually bring about improvement and not summative stopping the learning, which it does if you spend too much time on it, so the foreground, the formative; background, the summative.

Skip to 16 minutes and 51 secondsDYLAN WILLIAM: And I think the work of David Yeager and what he calls "wise feedback" is very relevant here. He actually worked with some students in New York who'd written an essay about a personal hero. And half the students got their work back with some comments from the teacher, accompanied by a note that says, "I've given you some feedback, so you have some comments on your work." I think it was the other way around. "I'll give you some comments, so you have some feedback on your work." And the other half of the students got a note that said, "I've given you some critical comments on your work because I have very high standards.

Skip to 17 minutes and 22 secondsAnd I believe that you can reach them." And for African-American students in the control group, only 25% of the students chose to resubmit their work a week later. Whereas for the students who got the more positive comment note, 75% chose to resubmit. It was a really dramatic effect. And so what we're seeing is that a lot of students don't understand why they're being given feedback. So as Chris said, everything-- anything you could do to actually get students to understand that the purpose of feedback is to help students do better in the future, than the more likely students are to react positively to it. To use Doug Reeves's metaphor, it's either the medical or the postmortem.

Skip to 18 minutes and 1 secondAnd too many students get postmortems. Because as I keep on saying, the purpose of feedback is not to improve the work. It's to improve the student. And too often, students think that the purpose of the feedback was to just do corrections for the last piece. And I think that's completely irrelevant, unless it helps them do a better job for a similar task at some point in the future. So that idea of feedback or marking-- as Chris now bans her students from saying-- is to improve the student. It's not to improve the artefact the student has produced.

Skip to 18 minutes and 33 secondsANDREA MAPPLEBECK: That's lovely. Thank you both. Some really rich ideas already. And we're only halfway through the questions. That's fantastic. I forgot to say to people at the start-- I've even written it on my notepad-- that having a piece of paper and a pen ready is always really helpful when we have our Q&As. So thank you both so much. Moving on to our next questions, which we put under the category of classroom culture. Chris this is a question that was raised by several people. It's about-- because one of the things we've talked about very much in our Planning for Learning online course is this idea of the power of errors and the opportunity for us to learn from it.

Skip to 19 minutes and 12 secondsSo people have been asking us, how can we create this learning environment where our learners, our students, feel safe to make these mistakes? So what ideas can you share with us for that one, Chris?

Skip to 19 minutes and 22 secondsCHRIS HARRISON: Well, I think you've got to focus on improvement rather than calling them mistakes because they want to know-- you need students to know why they're doing a particular activity. So one of my favourite ones that I saw in a classroom was, again, working with 17-year-olds. A social studies teacher set them a homework task of writing an essay on some aspect of the curriculum that they were focusing on. And the next day when they came in, when they brought their essays in, she just gave them three essays that she claimed were written by other students.

Skip to 20 minutes and 0 secondsI know she'd mocked them up in various ways so that they actually had some of the sorts of errors or some of the maybe missing parts compared to, say, a perfect actual answer. And what she did was to ask them to write them in terms of the best down to the worst of these three particular essays and to talk about the reasons why one was better than the other. And this took about 20, 25 minutes of talking in groups and then talking in a whole class situation with the teacher. And then she asked them to get out their essays that they've done themselves. And they had to then slide their essays into this group of three.

Skip to 20 minutes and 41 secondsWas it better than the number one? What it worse than number three? Was it between two and three? Where was it in this scale that they'd actually created? And then they had to justify that to a peer. And a peer had to read their essay as well. When they'd actually done all of that, she then gave them the criteria that the examination board had produced. And it was quite interesting how they picked up on quite a lot of the ideas that were in the original mark scheme for that essay but had actually done it more subtly and understood much better.

Skip to 21 minutes and 11 secondsAnd she then gave them the option of either handing in their essay they'd done the previous night, or they could actually take it away and do any improvement on it they wanted, having had feedback through this iteration of activities that they'd done, and hand it in later that week. And what was interesting, only two students out of a group of 15 handed their essay in. The other 13 took it away and decided to make some improvement on it. So it's this thing really of it's very hard in your own work-- particularly in something like an essay, the way you put a lot of effort and time into it-- to see areas where you might improve.

Skip to 21 minutes and 47 secondsBut when you actually take them through activities like this teacher did, it highlights for them not just where they need to take their own work and make improvements on it, but they see actual good examples in other students' work that they can then incorporate. In other words, they're learning as they're actually doing it.

Skip to 22 minutes and 6 secondsSo if you can create this sort of atmosphere where they realise, one, that they can do things better-- and making a second attempt might make things better-- two, that they can use one another as a resource-- I really rate the work of Margaret Heritage, who talks very much in the American context how she sees students using one other as a resource-- and, three, that they actually come to feel that improvement is what's important and not just, I've done it, so that's it. It's ended. I cannot-- Learning from it or learning from the activity is the important part of this type of activity.

Skip to 22 minutes and 49 secondsDYLAN WILLIAM: I think the important thing in what Chris was describing in that teachers work is that the teacher had been very smart in separating out the understanding of the criteria for success from the students applying it to their own work. So they were understanding what made good quality work in the context of anonymous peers work, which is much less emotionally charged. And therefore, once they'd understood those kinds of criteria for success, they'd be able to apply them to their own work, partly because it's less emotionally charged, but also probably because the cognitive load is reduced because you're not having to internalise the criteria for success and monitor your own work at the same time.

Skip to 23 minutes and 31 secondsAnd I think those things are very important. In terms of the general thing about mistakes, I go backwards and forwards about this word "mistakes" because, clearly, it has a negative set of connotations. And I'm really not sure whether to avoid it or to try to give it a more positive meaning. So every student I've ever met understands that if you're serious about sport, for example, and you want to get stronger, you need to be lifting weights that are hard for you to lift. And yet, many-- the same students will welcome it when their math teacher gives them a set of easy sums to do.

Skip to 24 minutes and 9 secondsAnd so, for me, it's about getting students to understand that if there's not some difficulty involved, if you're not causing a bit of struggle, it's probably wasting your time. And I say to students, mistakes are evidence that the work I gave you is tough enough to make you smarter. And so I say to students, if you're not making mistakes, I apologise. I'm probably wasting your time because this is probably too easy for you. And this is where I think Carol Dweck's work on growth mindset is so important. There's a big debate right now about whether this is a big idea or a trivial one.

Skip to 24 minutes and 40 secondsAnd a lot of research studies have found that Carol Dweck's mindset doesn't change student achievement very much. But I think that's missing the point because just changing growth mindset from fixed to a growth mindset has a very small impact, maybe a fifth of a letter grade. But what having a growth mindset does is make you much more receptive to feedback. So, for me, growth mindset is not an end in itself. It's a means to an end. It's a means to making students more open to getting feedback about how their work could be improved.

Skip to 25 minutes and 15 secondsAnd so this, for me, explains the research results we're getting, which is that where feedback is given, then you might get a positive effect from growth mindset interventions. But if you're just changing mindset and not doing anything else, then you're not going to get much of an impact on the student achievement. So it's about a classroom where students understand that the work that makes you struggle, that makes you have to think, means that you're learning more effectively. And I think just talking to students all the time and then using that really important-- the most important word in every teacher's vocabulary, which is "yet." When the students say, "I can't do this," you just say, "yet."

Skip to 25 minutes and 49 secondsAnd with young children, I say to them, remember how hard it was to button up your coat when you couldn't button up your coat and how easy it is now? Remember how hard it was to tie your shoelaces when you couldn't tie shoelaces and how easy it is now? And just reminding students constantly of the fact that they're improving, I think, is a way of creating this idea that if it's causing you to make mistakes, if it's causing you to struggle, then it is probably worth your while because it's going to make you smarter.

Skip to 26 minutes and 15 secondsANDREA MAPPLEBECK: Thank you both again. Really helpful ideas there. I think one of the powers of having you two as our experts here is the fact that you often reference for us articles and researchers who are out there, which gives our learners on the course the opportunity to go away and deepen their understanding too. One thing I wanted to pick up actually-- because I know that I'm trying to make notes of who those different learners are-- Chris, you mentioned earlier about a project that your teachers have been working on over time. And I didn't catch the name of that project. I don't know if you mentioned it and whether it would be useful just picking up. Now I remember.

Skip to 26 minutes and 50 secondsCHRIS HARRISON: Sorry, I don't think I gave the name. It's called, Assist Me.

Skip to 26 minutes and 54 secondsANDREA MAPPLEBECK: Lovely. Thank you. That's really helpful. So I just think-- [INTERPOSING VOICES]

Skip to 27 minutes and 0 secondsANDREA MAPPLEBECK: --that you two developed for us to use. And it's really powerful for people to come back to. So thank you for that and for all of those references that you've just given to the answer about developing this learning environment because, obviously, that's something that's going to take time. So thank you both. Moving on then, thinking about classroom culture. The next question, again, misconceptions were discussed in depth on the calls on this Planning for Learning Course. And several teachers have raised the issue that they were worried that if they are upfront and explicit about misconceptions, they may actually reinforce them in their own students' thinking.

Skip to 27 minutes and 36 secondsSo the question here to you first, Dylan, is should we nip the misconceptions in the bud to avoid this risk of reinforcing them in our students' thinking?

Skip to 27 minutes and 45 secondsDYLAN WILLIAM: Well, I think the first thing to say is there's some misconceptions that you can probably prevent students from acquiring if you plan your teaching very carefully. So, for example, in mathematics, if you only show students triangles with the horizontal line at the bottom, then students will think that the orientation of the shape is part of the shape name because that's what we do in ordinary language. In America, they play baseball at a diamond, which is in fact a square. But people think of it as a diamond because you're standing usually at home plate. So in our natural language, we tend to use shape words to connote both shape and orientation.

Skip to 28 minutes and 25 secondsSo making sure that you're giving students different kinds of triangles, different orientations. If you're using the word trapezium, make sure you understand that-- make sure that it's not the traditional trapezium, which is often symmetric. But I think that there's no convincing evidence that misconceptions have an important role to play. And this is a debate that's been going on in psychology for about 50 years between a behaviourist approach, which is to say, let's just design our teaching really carefully. We're not going to use misconceptions because students might run with the misconception rather than the correct idea. The constructivists said, look, students are going to make sense of this in their own way.

Skip to 29 minutes and 2 secondsYou might as well deal with the misconceptions, expose them, and talk them through. And I think that argument has been pretty fruitless for a long time. But the evidence is beginning to pile up that the associationists, the behaviourists, were wrong. And the constructivists were right. In other words, misconceptions are inevitable. And the struggle, the productive struggle, of figuring out why this is a misconception is actually really important in terms of remembering how to think about this correctly. One more thing to say, in sitting science, there's some evidence that these misconceptions never go away. So when I'm grabbing a cold door handle on a cold day, it really feels as if the cold is coming into my hand.

Skip to 29 minutes and 44 secondsBut then my training as a physicist kicks in. And I say, nope, what's really happening here is the metal is conducting the heat away from my hand. So I think that discussing these misconceptions, making students grapple with the difficulty, why this way of thinking about this is more powerful than another way of thinking about it, it might not be more correct. But it's more powerful in terms of scientific reasoning for giving students more powerful models to think with and then understanding why the intuitive way is often less powerful. I think that's a really important part of effective teaching, particularly in mathematics and science, where these misconceptions are so easy to acquire because of our experience with the world.

Skip to 30 minutes and 28 secondsCHRIS HARRISON: I certainly think nipping it in the bud is not the right way to go about it from a science perspective. I see quite often with inexperienced teachers how they try and do that. So they ask students about what's happening in when, say, an organism is moving. And they say, oh, it requires energy to move. And therefore, it requires breathing. The teacher goes, no, no, no, you mean respiration. So they just correct the word. And so the students start to use them. And they don't know whether it's going to be respiration or breathing. They just wait until the teacher corrects it. And they have a guess. It's a 50-50 chance.

Skip to 31 minutes and 11 secondsThey don't think it through about what's actually going on. And part of the problem is that a lot of the ways that particularly science is taught these days is it's very compartmentalised. And when it's compartmentalised, then those misconceptions tend to stay within those compartments. So they don't think broadly enough. And they don't think things through. And they don't talk things through. So getting students to have a more dialogic approach in the classroom, where they're not just answering the word respiration or breathing or where they're not just-- when Dylan was talking about getting hold of the cold door knob-- they're not just talking about, oh, this is conduction, and which way is the heat flowing.

Skip to 31 minutes and 52 secondsSo it's not just done in a factual way. But they actually are given opportunity to talk about it. One of the best ways that this is done within science is using Keogh and Naylor's concept cartoons, where they have some sort of phenomena. And the snowman with a coat on is the one that everybody knows. And then they have kids with speech bubbles above them sort of making their comments about what they think is going on. And the important thing here is for the teacher to say to the class when they're looking at this, what do you think? Talk with one another. What do you think? Again, it's the way it's used in classrooms.

Skip to 32 minutes and 26 secondsWith some teachers, they say, what's the right answer? As though one kid's got it right, and another kid's got it wrong. And this is when the misconceptions just tend to stay in those kids that's got them. And the ones that haven't got the misconceptions, sometimes they're fine. But sometimes they get pulled into the misconception area as well. So the important thing is to have dialogue to get kids to think, to go through those phenomena many times in many different contexts because they're not easy science. Thinking in a scientific way is not easy. And sometimes-- just as Dylan says with his triangles-- sometimes our textbooks lead them to misconceptions anyway.

Skip to 33 minutes and 4 secondsWe had a really interesting one in photosynthesis a couple of years ago, where we realised the kids actually thought that chlorophyll was a magnet because on the diagrams, the arrow goes straight into the leaf from the sun. So the leaf must be pulling the light in from somewhere. And one of the kids were saying, well, yeah, it has got magnesium at the centre, as though magnesium was a magnetic metal. So we had all sorts of misconceptions coming there. Some of those, the teacher decided to sort out there and then. The fact that magnesium is not magnetic didn't matter. She left that for another time.

Skip to 33 minutes and 39 secondsBut she did start talking to them much more about what the word absorb meant and that it wasn't the same as the word attract. And that was the beginning of it. She then picked it up again later when they were doing work where they were looking at photosynthetic rate and size of leaf. And she talked again about the word absorb and what it meant and tried to explain that the diagrams they saw in their book was just a shorthand way of doing it. And in actual fact, the leaf was not a magnet. It was just the leaf was hitting-- the light was hitting the leaf's surface.

Skip to 34 minutes and 12 secondsSo, yes, I think having more dialogue in the classroom really helps, particularly in science classrooms. And I think that realising you're going to have to come back time after time and address these possible alternative views from the students is important. You're not going to solve it by just correcting it once, which, unfortunately, inexperienced teachers sometimes think.

Skip to 34 minutes and 38 secondsDYLAN WILLIAM: And I think that's it's really important to reinforce the point that Chris made about sometimes when we correct students, all we do is change vocabulary. So, for example, I remember talking to a child about what, when you have sugar in water, and you stir it, what happens to the sugar? And the child says it disappeared. And then the teacher says, no, it's dissolved. And so now you think that the students understood it because then next time, what happened to the sugar? They say it's dissolved. But it could be that all you've done is convince the child that when they're talking about sugar and water, you replace the word "disappear" with the word "dissolve."

Skip to 35 minutes and 14 secondsSo where did it go when it dissolved? Oh, it disappeared. Why does the water still taste sweet? Because it left the taste behind. So this, I think, it really emphasises the point that Chris made about the need to get students to talk these ideas through because then you can begin to understand where it is-- basically, students do not learn what we teach. And you can't figure out how to teach them well unless you find out where they are. And to do that, you need evidence. You can't look inside a child's brain to find out if the concepts are correctly arranged. You need to get them talking. And that's why dialogue is so important.

Skip to 35 minutes and 51 secondsAnd one more thing that viewers of this video might be interested in, there's a very good link here to Daniel Kahneman's work about System 1 and System 2. So he actually says that our-- the way we think about things, we have this intuitive way of thinking about things, like the idea that the door handle is cold. And the cold is coming in. And then System 2 is the more kind of mature, developed, thoughtful system-- ah, no, that's not what's happening. And so I think that might be quite helpful to science teachers to think about this, that we naturally think intuitively and immediately. And then we need so often to correct this.

Skip to 36 minutes and 28 secondsAnd it may be that conceptual change-- which is something that scientists and science educators have researched for many years-- it may be that conceptual change never really happens. What happens is that students learn that when you're in a science context, you need to think with scientific models. And in other contexts, other models are equally appropriate. So it's important to think about, what's the right way of thinking about this in this context? What's the best model? And I think the idea of science as being modelled-based reasoning is a very helpful way of thinking about this because it also then says to students, the models we've got now are great for the things we're doing right now.

Skip to 37 minutes and 5 secondsWe get students saying, but I was told this differently when I was younger. And so students often think that they've been lied to because simpler models they were told when they were in year 7 are no longer sophisticated enough for year 11 work. And this idea that these are models-- and as George Box, the statistician, said, "All models are wrong. Some are useful." And I think that's a very powerful way of helping students understand why what we're doing in science in particular is using more and more sophisticated models. And that's why sometimes this knowledge is provisional. And we'll have better models in the future.

Skip to 37 minutes and 41 secondsAnd choosing the right model to think about this in the right way is itself something that's well worth practicing with our students.

Skip to 37 minutes and 50 secondsANDREA MAPPLEBECK: Thank you both very much. That's really helpful. So lots of really good ideas there about developing our classroom culture and thinking about this learning environment. So we've got two last questions. The first one comes from June. And June is teaching students where English is not their first language. And one of the things that she's particularly interested in finding out your ideas about is how she can get their prior knowledge when they don't understand the native language that she's actually building and working in, in particular for approaches to assessment for learning with her students? So, Dylan, do you have any ideas that might help June with this?

Skip to 38 minutes and 29 secondsDYLAN WILLIAM: I think at the outset, it's really important to acknowledge that this is really, really difficult. Students, if they don't have the language to express their insights, then it's going to be very difficult for the teacher to find out what's going on. And June has mentioned that she gets kids to do videos or diagrams. If you've got other students in the same classroom who speak the same language as the student, then often that can be a stepping stone. But I think that it's important to acknowledge that this is really just about the hardest aspect of formative assessment to do well because there's just no common ground. And so it's very hard for students conceptually to build a cognitive load.

Skip to 39 minutes and 13 secondsIf that student isn't fluent in the language they're trying to talk to you in, then they're going to be spending a lot of their mental resources on finding the right words in the foreign language, which leaves less for thinking about the conceptual material, the maths or the science. So I just think it's important to acknowledge at the outset, this is really, really difficult. And it will only come as a student's language becomes more automatic. And therefore, they're able to think more scientifically. And I think what I would say is if a student can't express their insights to you in the language that you can understand, don't assume that they don't get the science or the maths.

Skip to 39 minutes and 53 secondsIt may be that their language is inadequate to express to you the insights that they have. And so that's why diagrams, pictures, can be quite helpful ways of students externalising their thoughts.

Skip to 40 minutes and 8 secondsLovely. Thank you. Chris?

Skip to 40 minutes and 11 secondsCHRIS HARRISON: [INAUDIBLE] Dylan. I think it is a difficult area. And it's not an area I'm expert in myself. I tend to go to either Margaret Heritage and Alison Bailey's work. They're putting a lot in this area in the States. Or in England, Constant Leung, who's from King's College London, actually has a big project looking at this across schools in London and the south of England. But I think, just as Dylan said, allowing the students different ways of actually expressing their ideas, not just through language or as they're trying to develop their language, but finding ways for them to explain the science, the geography, whatever it is, through different modes.

Skip to 40 minutes and 54 secondsBut also stimulus material coming in in a way that can actually allow them to start at least thinking about it. And then the teacher doing their best to really tap into their thinking. That's the hard bit because the teacher hasn't got the language skills probably in order to find out really what the student's doing and thinking. So it's trying to boost that communication part of the actual event that's important. But, yeah, I agree with Dylan. It's quite a difficult thing to do for teachers. However, can I just say one thing?

Skip to 41 minutes and 27 secondsOne thing I always tell my pre-service teachers and that is that despite diversity meaning sometimes it makes your job harder in that you've got to deal with a range of languages in your classroom, it actually makes it richer because the children are bringing in a wider range of experience than you might have in a more monoculture-type classroom. And I think that's important to actually acknowledge. The classrooms that I see these days are just rich with ideas because the kids come from so many different environments. So the language might be a problem. But then there's other aspects of it that actually upset that, that I actually think makes it easier to teach. So it's that balance between those two aspects.

Skip to 42 minutes and 12 secondsDYLAN WILLIAM: Just to reinforce that, I remember hearing a talk by Robert Swan. And he's a polar explorer. He's the first man to walk to both poles. And he was talking about problem solving in adverse conditions. And he said something that's really stuck with me ever since I heard him say this. He said, "When everyone is thinking the same thing, no one is thinking." And that really reinforces the value of diversity. Because when students are expressing their scientific or mathematical ideas, then if you disagree with that student, then you're listening in a much more engaged way because you're trying to find out ways of combating that or finding flaws in the other person's argument.

Skip to 42 minutes and 53 secondsSo then, as Chris says, diversity, I would say, is a teacher's most valuable instructional resource. Diversity increases engagement, gets you better discussions, gets you better ideas, and therefore, I think we should often seek to create diversity even when it's not there. So I often ask students quite vague questions because I quite want-- I quite like them to think about these things in different ways, which then gets us much better discussions because you have different points of view.

Skip to 43 minutes and 24 secondsCHRIS HARRISON: Yeah, I do that with my pre-service as well. So when I'm getting them to think about what's a good question to ask in a particular activity, and then we look at the questions they come up with, we judge how good the question is by thinking, well, what might your high-attaining student in your class answer? What might the low-attaining student who needs support answer? What might that weird child in your class answer? Because every class has a particular child who thinks outside the box, the one that doesn't think the way most of the kids do. And if you get three different answers to your question, you know that's going to be a rich-dialogue question.

Skip to 43 minutes and 59 secondsIf you get the same answer from all three kids in your head, then you know it's just going to be one of those questions that happen in class. Find another one that's going to stimulate thinking and talking better.

Skip to 44 minutes and 11 secondsANDREA MAPPLEBECK: Lovely. Thank you both very much. It makes me smile listening to you two. Right, moving on to our last question then. Chris, I'm going to ask you first about this. This is a question from Meagan, who has been working on the course and found it purposeful and reflective and helpful. And they want to take it forward and create an observation checklist to support reflections happening in their lessons by thinking about some look fors. So if the students are doing, dot, dot, dot, the teacher is-- And wondered if you have suggestions. And then Meagan also goes on to say that they want to use this for themselves as well in supporting colleagues in professional learning groups.

Skip to 44 minutes and 47 secondsSo I thought you might have thoughts about a checklist but also thoughts about our teachers working with others once they've finished our online course.

Skip to 44 minutes and 57 secondsCHRIS HARRISON: OK, well, I don't like checklists. But I don't mind having some things to look out for. And certainly when I go into classrooms, there's a number of things I look out for to try and get a first assessment of how well the classes are working in and ready to do formative work. So the three things I look for is one is what's the dialogue like? So are the kids asking questions or is the teacher asking all the questions? How much talk is going on? Have the students got the language to talk both about their learning and about assessment? Are they respectful as they talk to one another? So they are listening to one another.

Skip to 45 minutes and 40 secondsSo it's the behaviours of the students in terms of dialogue and the behaviour of the teacher. Is the teacher actually trying to reduce her talk and listening to the student talk? So that's the dialogue side of it. The collaboration side comes in with that. So how are kids working with one another and in groups and in whole class situations? And then the final thing really is I look for-- I like to talk to individual kids and talk to them about their learning to find out, just ask them, well, what have you learned today? Can they answer that? And what's different now from what was there in your head at the start of the lesson?

Skip to 46 minutes and 22 secondsAnd can you show me how you've improved in a particular area? And certainly, many classrooms I go into, you get a really good feedback from the students and from just the atmosphere that's there. You can pick up and understand that formative assessment is actually happening well within that classroom. I think the other thing I look for is actually outside the classroom. So when I've been watching a teacher and her class, what I like to do is to then get the teacher to talk me through how she saw events in the classroom because, of course, she's got all the back history of the previous lessons and the students and they're learning and what she's done with them before.

Skip to 47 minutes and 4 secondsAnd can she tell me why which events were important in terms of students learning? Both in terms of her collecting evidence of the learning, but also collecting evidence of future needs of those students and what she's going to plan to do next. So you don't have to necessarily go and observe it. Sometimes you can just get teachers to talk through what happened and why they're going to do a particular approach as they move forward with the student learning. So there's lots of things you can pick up on. And it's really just honing in to find out what's going to help that teacher and that class at that time.

Skip to 47 minutes and 42 secondsI don't think you need to collect evidence or try and do all of that. It's just finding out which will work best for you and your colleagues in that particular situation. And often it's just chatting about an event that happened in the lesson and explaining why it's important because it pushes the first teacher into reflecting on it and to making some decisions. But at the same time, the other teacher will get some idea of both how formative assessment has worked in the other teacher's classroom and how it might work in their own because we always tend to do that. When we look at somebody else teaching, we think, how might we do it with our class?

Skip to 48 minutes and 19 secondsSo it's learning from one another in that way.

Skip to 48 minutes and 22 secondsANDREA MAPPLEBECK: Thank you, Chris. Dylan, anything to add?

Skip to 48 minutes and 27 secondsDYLAN WILLIAM: I think I'll send you a checklist that we have actually developed as kind of a memoir for things to look out for in classrooms. It's not a-- you wouldn't see more than two or three of them at any one lesson. But it's a list-- if formative assessment is happening well, you'll probably see some of these things. But I think I would start with this idea that formative assessment involves a pedagogy of engagement and a pedagogy of responsiveness. So the first question I look for is who's participating? Chris will remember, one of the teachers in the original KMOFAP project described his classroom discussions as a conversation between him and six or seven individuals, witnessed by 24 sleepy onlookers.

Skip to 49 minutes and 12 secondsAnd so the question is, who's participating? Who's contributing? Who's engaged? That's seems to be very important. And then, how is the teacher responding? And one of things I like to look for is a nice idea from Brent Davis of the University of Alberta. He contrasts evaluative and interpretive listening. So when a teacher asks a student a question, the teacher who listens evaluatively is just listening for the correct answer. And you can spot those teachers because they say things like, "nearly," or "close," or "almost." But the best teachers don't listen evaluatively. They listen interpretively. They're listening for the meaning in what the student says.

Skip to 49 minutes and 48 secondsHow can I figure out what that student is thinking by listening carefully to what they just said? And so that's something that seems to be the hallmark of really good formative assessment practise is teachers listening for what they can learn about the students thinking by tending carefully to what that student is saying. And therefore-- and then, obviously, making the right kinds of adjustments to what they're talking about. And then, as Chris said, also then talking to the teacher. Did you use evidence to make decisions that took you away from your original plan? Was there a change of course?

Skip to 50 minutes and 24 secondsThose kinds of things are very interesting as evidence of the teacher's use, of evidence of, I used that evidence to make a decision. I changed my mind. I realised this wasn't going very well. And one of the things I often also ask my pre-service teachers is, once they've planned a careful lesson, what would cause you to give up on this lesson plan? Because one of the things we see is novice teachers often persist with a lesson plan even when it's clear that it's not working very effectively. So this whole idea of, what evidence would make you think maybe a different tack is needed?

Skip to 50 minutes and 57 secondsSo those kinds of things, lessons, changing tack, and student engagement, and teachers listening carefully to what students are really saying, not just listening for the correct answer.

Skip to 51 minutes and 8 secondsANDREA MAPPLEBECK: Lovely. Thank you both so much, as always. We get through so much in a short amount of time. And you produce a feast for us of ideas for us to think about. I've been scribbling and colour coding as usual on my pad, something that I try and do. Lots and lots of rich ideas that have come out from you. I think it's really interesting, we gain so many teaching ideas that you share with us. I think one of the things that's really powerful is how you anchor those to exemplars. So you give us ideas of teachers in classrooms, which is really interesting because that's one of the things that we do on the course.

Skip to 51 minutes and 44 secondsAnd I know that some of the ideas you've talked about we've actually pulled out on our courses, the ideas about rich dialogue, intentional dialogue, rich questions, the idea of students ranking, the power of exemplars, using misconceptions, concept cartoons, science being a model-based reasoning approach, so lots and lots of really good teaching ideas. As always, you've given us food for thought and things to go away and read upon. So we've heard lots of different researchers mentioned because I know that you two have so much that you do. You've mentioned a tiny bit that you both do, your Assist Me project, Chris. Dylan, your checklist, that would be fantastic to have.

Skip to 52 minutes and 22 secondsBut we've had David Yeager, Alfie Kohn, Margaret Heritage, Dweck, so many different people who you've referenced for us to go to think about. I think, for me, the themes that have come out tonight that are really powerful are some things that can be tweaks. In particular to our language, you've mentioned talking about guidance rather than marking. You've talked about thinking about improvements rather than mistakes. You've talked about not looking for answers but asking for what you're thinking so that we're interpreting. But, for me, they might seem like small things. But they're huge things that actually can shift a teacher's beliefs and their practise that can have huge impacts on what's happening in the classroom for our learners.

Skip to 53 minutes and 3 secondsAnd the things that have come out that I think are really powerful that resonated for me throughout the discussions that you've raised is this idea of purposefulness. And you talked about it in terms of evidence collection. You talked about in terms of recording the evidence. You've talked about in terms of planning. You've talked about in terms of feedback. You've talked about in terms of improvements for next steps. And what then it all resonates and, for me, it all boils down to is something that you've both said, and that's putting learning at the front and centre. It's not about task performance.

Skip to 53 minutes and 37 secondsIt's about students developing their cognitive capabilities in the subjects that they're learning and how they do that on their way. And the powerful way to help them with this is using assessment continuously alongside that learning and feedback and to look at improvements in the learning alongside that too. And, for me, it's just been a really rich and powerful opportunity to pick your two brains to help us keep thinking about how small tweaks to our thinking can have big impacts in our classrooms. So, as always, thank you both to you both so much. I know that you're in different parts of the world. So you'll be in different time zones.

Skip to 54 minutes and 15 secondsSo thank you for making the time to actually-- for us to meet together. Thank you, as always, to participants on the calls for their reflective ideas and for the questions that they've raised for us today. Thank you, as ever, to the National STEM Learning Centre for giving us this fantastic opportunity. And thank you to Matt in the background for supporting us. As I've said, there's lots of ideas that Dylan and Chris have talked about that are in the course. So if you're coming to the course through the Q&A, please do go and look at the ideas exemplified by teachers in the classroom. So thank you all very much for your time and ideas.

Q&A with course educators

The Q&A sessions on courses from the National STEM Learning Centre provide you with the opportunity to ask more about the course content and issues from your own classroom practice.

Dylan Wiliam, Chris Harrison and Andrea Mapplebeck recorded their answers to a selection of your questions. We’re particularly grateful to the educators who made time for the Q&A alongside other commitments and travel as we recorded across three continents!


  • Decision-driven data collection
  • Recording of data and evidence
  • Summative assessment
  • Classroom culture
  • Assessment for learning and language barriers
  • Reflecting on teaching

This is an open step, so you can bookmark the URL to your favourites and return to it at any time. We will also upload the video to STEM Learning YouTube channel.

Please note: if you post a question here it may be featured in the video recording along with your first name. The recording will be publicly viewed via this step and may also be uploaded to the STEM Learning YouTube channel.

Share this video:

This video is from the free online course:

Planning for Learning: Formative Assessment

National STEM Learning Centre