Skip main navigation

How rare is replication?

In this video, Ted Miguel discusses how rare replication is published economics research.
Despite how valuable your replications are, it’s very rare for replication to happen. Now Hamermesh’s analysis is already out of date because I think things are starting to change. But, for the most part, replication is quite rare. And, you know, basically what he shows in the papers – Even a bunch of years later, after publication, most empirical papers have not been replicated in any way that is sort of documented in the research record. So this is the table. The first two journals are, good field journals in labor economics. Industrial Labor Relations Review. And, then the Journal of Human Resources – JHR. Those are two good field journals.
And then the AER, which is like the flagship American Economic Association Journal – the most cited journal, etc. He basically reached out to the authors who had written – All the authors who had written empirical papers in these journals. He got really good response rates. He is a pretty prominent scholar. He reached out to them. You know, all they had to do was answer a really simple question. “Did anyone request your data?” basically. So, most people replied. So, it’s probably pretty reliable. And for the Field Journal articles, four years later, three years later, for the median article, no one had asked for the data. So the median article no one had sorted of checked.
For the AER, the median was three requests. And this was like six or seven years later. So, again, these are by definition, all the quite influential articles. They are very well cited. Very rarely is the data requested. In fact, many of these requests weren’t even for replication activities. Very often people request your data, as you’ll find when you start collecting more data; They request your data because they’re like, “I really want this variable. I’m doing this other analysis. I want that variable.” So it isn’t necessarily even for a replication exercise. This is like an upper bound of some kind on how many times people were looking to replicate.
And this is now the distribution of requests, again, for the Industrial and Labor Relations Review. There we go. And Journal of Human Resources. And, again, you can see in most cases it’s zero. And in a small number of cases, one or two people have requested. It is a pretty skewed distribution for some of the probably most influential papers. You see a few papers where there were a lot of requests for data. So, I guess that conforms with what we’re saying. And again, if you have this kind of more optimistic view, like, “Look, for the most important work, they will get replicated. Maybe these are the best papers.” Well, then maybe, you know, you’re not as worried.
“Economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practised. Why is this?” asks Dr. Hamermesh in his Viewpoint paper on Replication in Economics. If replication is so important why are published replications so rare? In these next two videos, we discuss just how rare replications are in economics journals, as well as possible reasons for this scarcity. Dr. Hamermesh ends his paper asking how we can promote replication and puts forth some potential solutions.
This article is from the free online

Transparent and Open Social Science Research

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education