Skip to 0 minutes and 0 seconds Despite how valuable your replications are, it’s very rare for replication to happen. Now Hamermesh’s analysis is already out of date because I think things are starting to change. But, for the most part, replication is quite rare. And, you know, basically what he shows in the papers – Even a bunch of years later, after publication, most empirical papers have not been replicated in any way that is sort of documented in the research record. So this is the table. The first two journals are, good field journals in labor economics. Industrial Labor Relations Review. And, then the Journal of Human Resources – JHR. Those are two good field journals.
Skip to 0 minutes and 38 seconds And then the AER, which is like the flagship American Economic Association Journal – the most cited journal, etc. He basically reached out to the authors who had written – All the authors who had written empirical papers in these journals. He got really good response rates. He is a pretty prominent scholar. He reached out to them. You know, all they had to do was answer a really simple question. “Did anyone request your data?” basically. So, most people replied. So, it’s probably pretty reliable. And for the Field Journal articles, four years later, three years later, for the median article, no one had asked for the data. So the median article no one had sorted of checked.
Skip to 1 minute and 13 seconds For the AER, the median was three requests. And this was like six or seven years later. So, again, these are by definition, all the quite influential articles. They are very well cited. Very rarely is the data requested. In fact, many of these requests weren’t even for replication activities. Very often people request your data, as you’ll find when you start collecting more data; They request your data because they’re like, “I really want this variable. I’m doing this other analysis. I want that variable.” So it isn’t necessarily even for a replication exercise. This is like an upper bound of some kind on how many times people were looking to replicate.
Skip to 1 minute and 48 seconds And this is now the distribution of requests, again, for the Industrial and Labor Relations Review. There we go. And Journal of Human Resources. And, again, you can see in most cases it’s zero. And in a small number of cases, one or two people have requested. It is a pretty skewed distribution for some of the probably most influential papers. You see a few papers where there were a lot of requests for data. So, I guess that conforms with what we’re saying. And again, if you have this kind of more optimistic view, like, “Look, for the most important work, they will get replicated. Maybe these are the best papers.” Well, then maybe, you know, you’re not as worried.
How rare is replication?
“Economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practised. Why is this?” asks Dr. Hamermesh in his Viewpoint paper on Replication in Economics. If replication is so important why are published replications so rare? In these next two videos, we discuss just how rare replications are in economics journals, as well as possible reasons for this scarcity. Dr. Hamermesh ends his paper asking how we can promote replication and puts forth some potential solutions.
© Center for Effective Global Action