Skip to 0 minutes and 0 secondsReplication is a central pillar of the scientific method and of the scientific enterprise. In other words, let’s say if somebody in physics comes up with a new result. They discover a particle, or they claim they have a new result, it isn’t really considered a real result until other labs can replicate it. Can do it over and over again. If it really is capturing whatever finding this lab came up with, is capturing some physical law or helping us understand some physical law, others should be able to replicate it. So, you know, the analogy isn’t perfect to the social sciences right off the bat.
Skip to 0 minutes and 41 secondsBecause, let’s say you find something in one group of people or one society, it’s not necessarily clear it will always hold elsewhere. We don’t have the same social laws or psychological laws or economic laws as we think particles do in physics. So, replication is always going to be a little different in the social sciences than it is in the physical sciences. And we might, just to start out, we should be resigned to the fact that we are going to have more heterogeneity in our treatment effect estimates inherently because of differences in the sample and the setting. Differences over time. People change over time.
Skip to 1 minute and 19 secondsSo even if you run the same experiment with the same people a week later, you may get a different result. So, that’s just a starting point. At the same time, the notion that a real result, a result we can believe in, is something that appears more than once in one data set or in one analysis – that still holds. If we are going to have confidence in a result, we want to believe that it holds a little more broadly than that. Hamermesh, in his article, makes a distinction between what he calls “pure replication” and “statistical replication.”
Skip to 1 minute and 50 secondsSo pure replication is literally the ability to take the same data and use the exact same analysis and literally just try to do exactly what the original authors did and get the exact same results. That’s basically what pure replication is. So, you know, you get my data. Unless I’ve made an error, you should be able to run the code and get the same results. And it turns out there are errors in the code. It turns out people make mistakes. Marty Feldstein wrote a very famous paper in 1974 that got a lot of attention claiming that when social security was rolled out – this huge social program – like the biggest social program in the US.
Skip to 2 minutes and 27 secondsWhen it was rolled out, it contributed to a massive reduction in private savings. And you might imagine why. If I know that I’m going to get a pension later in life, I may save less now because I know I’m going to get this pension or I believe I am going to get a pension. So I just spend freely now and I save less. But of course, in the aggregate, that could have big implications for growth if we’re sort of saving less, accumulating less capital, etc. And so it got a lot of attention. And Feldstein, of course, was the head of the Council of Economic Advisors for Ronald Reagan and was pushing this kind of agenda.
Skip to 2 minutes and 59 secondsIt turns out that this paper had a massive error in the construction of the main social security wealth variable. Like what you expected to get in social security benefits. And when you fix that error, the results go to zero. There was no effect that they could detect on private savings. That’s a big deal! This was like a top journal, one of the country and the world’s most prominent economist, writing something of huge public policy implications. For like six years this paper was floating around influencing policy. And if they hadn’t found the error, maybe it would have influenced policy more. Maybe, who knows what Reagan in the early ‘80s would have done if this finding was considered solid.
Skip to 3 minutes and 42 secondsMaybe they would have gone after social security. I don’t know. I mean, it’s possible.
Skip to 3 minutes and 48 secondsThis is a pure replication exercise that was really important.
Skip to 3 minutes and 54 secondsThe second type of replication – statistical replication, you’re testing the same hypothesis. You choose to use different methods. So if someone is critiquing a particular econometric approach in the minimum wage literature. And they say, “You know, this model – the model that’s used is just wrong. For various statistical reasons, this is wrong. I am going to use the more appropriate model.” That would be a statistical replication. We call it a “re-analysis.” That’s what we do. Now, in the other case, you test the same hypothesis in a different data set or in a different time – different setting. You might think of that as a robustness check. You might think of that as an extension.
Skip to 4 minutes and 32 secondsYou might think of that as a way to test external validity. But people use a lot of different terms for that. We’ll come back to this later. But there probably is somewhat more replication research going on than we think because a lot of that work is never called “replication research.” It’s just like, “Oh, I’m going to test the effect of the minimum wage increase on unemployment in Saskatchewan, Canada rather than in New Jersey. And I’m just going to say I did a minimum wage study.” I’m not necessarily going to call it a “replication.” But it kind of is a replication. You are trying to see if this hypothesis – I’m going to test this hypothesis in a new setting.
Introduction to replication
This activity consists of a series of videos that examine the different types of replication and the inherent differences between replications in the social sciences and those in the life sciences or physical sciences. It also introduces us to a viewpoint article by Dr. Daniel Hamermesh that discusses the importance of replication and how the scientific community can incentivize their publications. The next two videos focus on different aspects of Dr. Hamermesh’s paper.
© Center for Effective Global Action