Contact FutureLearn for Support
Skip main navigation
We use cookies to give you a better experience, if that’s ok you can close this message and carry on browsing. For more info read our cookies policy.
We use cookies to give you a better experience. Carry on browsing if you're happy with this, or read our cookies policy for more information.

Skip to 0 minutes and 0 secondsWhat should distribution of p-values look like? We’re gonna get into this with Card and Krueger. Card and Krueger’s paper is a meta-analysis. Sort of a study of publication bias in the minimum wage literature. Basic idea, and we’ll come to it with Card and Krueger. Is given a certain effect size, if you’re really well powered – so you have a bigger sample – for the same research design and the same effect, you should have more precise estimates. Therefore, t-statistics should be larger. That’s a pretty simple idea. And that’s going to be the starting point of Card and Krueger’s paper. They’re gonna ask whether this actually holds. Not in a literature with like three studies.

Skip to 0 minutes and 40 secondsThere’s gonna be 15 to 20 studies by the time they do their review in the mid-‘90s. So, a decent body of literature on a very important policy question.

Skip to 0 minutes and 51 secondsThey’re gonna be focusing on what happens to employment when the federal minimum wage goes up. Critical policy question. And what they find is – Their starting point is to say, well, there’s this kind of older cross-sectional literature of these 15 to 20 studies. And they tend to find these negative employment effects. But the more recent cross-sectional literature, using better research designs actually, tend to find pretty minimal effects. Either small or negative effects, or zero effects. I said, okay, why is that? One possibility is the research design is different. You think you’re estimating the same thing, but you’re actually estimating something different. That’s one possibility. Another possibility they bring up, which is interesting as well, maybe effects are changing over time.

Skip to 1 minute and 36 secondsLike maybe back in the ‘70s, when the first studies came out, there was a certain effect of the minimum wage, but now in the ‘90s it’s different. Like the economy changed. Other policies changed. The labor market changed. There’s a couple of reasons why you might see differences in different sets of estimates. But their preferred explanation, what they’re gonna come to and show a lot of evidence for, isn’t that at all. They’re basically gonna argue that there’s just like pervasive publication bias in the older literature. And really, the evidence isn’t consistent with anything else. So, some combination of publication bias and specification searching is going to lead to a very misleading literature on an incredibly important policy question.

Skip to 2 minutes and 16 secondsWhere leading scholars in the leading departments, are working on this for decades. And produced a couple dozen studies. Even in that case, they’re gonna say, “We really can’t trust this literature.” So that’s pretty important. This isn’t some esoteric sort of exercise. This is really important stuff. People are testifying in front of Congress about major economic policy changes. And based on 20 studies, they’re gonna say these studies do not have evidential value. Okay, so again, what we were just talking about, for a given effect size, and a given research design, if you double the sample size what’s gonna happen to precision? Precision is gonna go up by root two. Right? Straightforward. 41 percent.

Skip to 3 minutes and 0 secondsSo sample size goes up given the same design. T-statistic should go up. So, do they see that? When they have more data, do they see t-statistics that are twice as large? Well, this is the square root of the degrees of freedom. This is the absolute t-ratio. There’s just this really striking pattern which is most of these studies have t-statistics just above two, in these cases. They’re just this cluster of studies right over here with values of two. And there’s a couple that are off that line. When they do regress the logged t-stat on logged degrees of freedom, instead of getting a slope of one, they get a negative slope. So this is deeply problematic.

Skip to 3 minutes and 49 secondsIf it were zero, it would also be really problematic. It’s negative, so it’s like the folks out here, with big samples, given the same effect size – right? Big samples, should have bigger t-statistics. These guys out here just get zeroes. When you get lots of data, you get a zero. When you have a little bit of data, you just creep above a t of two somehow. You do a log specification rather than linear. You include four lags of past residential investment. You know, whatever. Whatever you need to do. That’s what you do. So this is a deeply troubling literature. It’s not one, it’s sort of negative one.

Skip to 4 minutes and 33 secondsHere’s another way of looking at the data.

Skip to 4 minutes and 37 secondsAgain, we’ve got the standard error over here, and then this is the estimated effect. And what they plot out is two times the standard error. And you just get this incredible clustering of studies right above two times the standard error. Totally unnatural. And then again, what’s really concerning here is these studies with really small standard errors down here, these are the well powered studies, big degrees of freedom and whatnot, these are kind of like the small, the zero effects. Like the small effects. So, those are probably the most reliable studies.

Skip to 5 minutes and 19 secondsThe bottom line is there’s this literature, 15 or 20 studies. Despite all that work, over a couple decades, it’s just not an informative literature. There’s so much just obvious, blatant publication bias in the literature, that we really don’t know what to make of it. And what they said is, “Look, there’s this recent cross-sectional study that’s well powered. With better designs, that shows zero effects. And that’s what we should be paying attention to.” And they’re like, “Oh, that’s our literature. Those are the papers we wrote.” And they were right.

"Time Series Minimum Wage Literature: A Meta-Analysis"

One way publication bias can have impacts beyond the scientific community is when poor policy recommendations are made based on skewed results. Such was the case when time-series minimum wage studies were used to make recommendations against raising the minimum wage in the United States. This video introduces David Card and Alan Krueger’s Meta-Analysis of these studies and demonstrates the effect of increasing an experiment’s sample size on the precision of its results.

Share this video:

This video is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley