Home / Teaching / Educational Leadership & Administration / Using Technology in Evidence-Based Teaching and Learning / Evaluating technology
Evaluating technology
Learn from two leading academics about the complexities of technology research evidence and hear useful advice for evaluating education technologies.
Hi both! How are you? Can you describe the Educate programme and why the need for it came about? Yes, certainly. I mean, it was conceived by our programme director, which is Professor Rose Luckin at the Knowledge Lab. Her work with EdTech companies and particularly looking at the needs of startups in EdTech really led her to conclude that there was a real lack of evidence-informedness to the EdTech ecosystem really. So when we look at the evidence of how technology has or hasn’t influenced the design of technology, sorry design of sort of resources for that have EdTech in them and the way in which they’re used in the classroom we had a really mixed bag of evidence that people were drawing on.
It leads to people sort of almost over-claiming on their EdTech without really having that thread of understanding about if it works, how it works, why it works, in which contexts it works and I think that’s something that both Kristen and I from our previous work had encountered and certainly drew me to work on the programme. I don’t know about you Kristen. Well, yeah, it was really interesting for me since I worked for a large technology company for many years and saw how pushing technology into schools when it’s been created for a different purpose, such as business, doesn’t always work and so you really want to have evidence of of why technology works.
You want to be able to communicate the evidence that already exists to schools to EdTech companies to create this kind of ecosystem of partners that communicate with each other. So for EdTech companies or educators it’s often difficult to get your hands on this kind of research and to understand what is high-quality research and what is not or what is applicable to your context and what is not. And to understand it, I mean academics tend to write for each other and they don’t want anybody else to play.
There’s a particular issue for small, medium sized companies when they start out which is they’ve got prototype ideas, they’ve got early developments on their tech; they want to go into schools or into other educational settings and see if they work and the schools are saying well, where’s your evidence? So it’s a chicken-and-egg situation where this is where elements are co-design and partnership come in. There’s a really interesting figure in the article that you guys wrote which is 44% of primary schools and 31% of secondary schools reported that EdTech implementation had helped them. Only 44% and only 33% is what I thought when I read that so why is it so atrocious and what can be done?
So I think one thing we haven’t mentioned yet is that EdTech is such a broad range of things. When it started out, it was really only computing but when you think now about the educational technology, you’ve got things that address the curriculum, you’ve got things that address aspects of teachers’ planning, you’ve got things that are directed towards assessment, you’ve got communication tools to parents or just internally within the school, and you’ve got admin stuff like you know timetabling and information management systems. Where we are now with EdTech is an awful lot of these things are integrated so you haven’t just got the complexities of each of those, you’ve also got the fact that they talk to each other.
Your assessment tool talks to your MIS and it talks to your parent communication and in order to come out and say, right this is really helping us, we can evaluate this in our school and we think it works. You need to be thinking about how you come to that judgement and often it’s a snap judgement. It is thinking about your first use of something, yes I liked it, I didn’t like it.
So one of the things that in our Educate for schools programme is supporting schools to really think through, right if I was going to pilot this in a slightly more systematic way, which just means I pin down which class and how long I’m going to use it for and what my key goals are and am I going to use all of it or a bit of it? Just trying to put some parameters around that little pilot so that you might still come out and say it doesn’t work but at least you’ve understood it doesn’t work in that context.
Absolutely and this is one of the things that we are working with schools on right now because it really is difficult for schools. Schools tend to hear about EdTech from other schools or other teachers and they think, oh well that worked in the school down the street or in the classroom of my friend so it’s bound to work for me. That, of course, is not often the case so we try to talk to teachers and schools about the basic tenets of what makes good research. What do they need to look for when they’re looking at a research study in terms of quality indicators?
So I’m putting a school leader hat on and a current educational climate, what are your views around… It’s entirely valid to say something doesn’t work but some of these things are quite expensive so the question I’m asking is, what should the approach be with regards to risk and trying something, realising this isn’t going to work and that there isn’t any shame around that? I mean, I think there’s some tools around the corner coming for this and at the same time there’s some strategies that we can learn from history.
One is certainly the work that Kristen and I are doing and going into schools is most of them are sitting on extensive amounts of technology that they a) they don’t sometimes even know they’ve got and b) they haven’t stopped to think whether they still need it and then sometimes, there’s technologies that do the same thing in the same schools. So there’s that spring clean and audit and our programme provides some audit tools to help schools to think through, right what have we got? What do we need? Have we got the things we need? You have to prioritise.
We can’t all do everything so actually looking at your school development plan, using some of the tools that we’ve developed, some of our partner tools. There’s an interesting new resource called EdTech Impact which is coming out soon. And other resources similar to that, that are going to help schools to do that self-evaluation. Then it’s about a sensible pilot evaluation that involves groups of teachers really drilling down on, does this really help us and in what way does it help us or not?
In this video, Iesha Small, Author, Consultant, and Innovation Leader interviews Dr Alison Clark-Wilson, Principal Research Lead, UCL EDUCATE, UCL Institute of Education and Dr Kristen Weatherby, Researcher, UCL EDUCATE, UCL Institute of Education.
Educate is a programme designed to support education technology companies in using research evidence to inform their products. Their work emerged from the observation that technology companies can often overstate what their product is able to achieve and this can lead to ill-informed decisions being made in schools about where a technology can be used and for what.
For some further practical advice from Alison and Kristen, you might like to take a look at their article for the Chartered College of Teaching about piloting and evaluating education technology in schools.
It’s worth considering how piloting and evaluation of impact can be addressed in the current context where schools may have been, or are still operating very differently due to COVID-19. It’s important to acknowledge that any online learning experienced at this time is ‘emergency remote teaching’ rather than ‘online learning’ and its impact and efficacy should be viewed through this lens. This article outlines this.
The video outlines how useful it might be to systematically pilot an education technology rather than making a snap judgement after a single use. Alison suggests identifying which class it might be trialled with, for how long and what goals you’d like to achieve by making use of the technology so that you can then evaluate how well it has met your goals. You may like to share with others how often systematic pilots are carried out in your school. How might they be useful when applied to education technologies?
When you have watched the interview and shared your thoughts in the comments, click the ‘Mark as complete’ button below and then select ‘Case studies’ to begin learning about how schools are implementing technology change in research-informed ways.
This article is from the online course:
Using Technology in Evidence-Based Teaching and Learning
Created by
This article is from the free online
Using Technology in Evidence-Based Teaching and Learning
Reach your personal and professional goals
Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.
Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.
