Want to keep learning?

This content is taken from the University of California, Berkeley, Center for Effective Global Action (CEGA) & Berkeley Initiative for Transparency in the Social Sciences (BITSS)'s online course, Transparent and Open Social Science Research. Join the course to learn more.

Skip to 0 minutes and 0 seconds What should distribution of p-values look like? We’re gonna get into this with Card and Krueger. Card and Krueger’s paper is a meta-analysis. Sort of a study of publication bias in the minimum wage literature. Basic idea, and we’ll come to it with Card and Krueger. Is given a certain effect size, if you’re really well powered – so you have a bigger sample – for the same research design and the same effect, you should have more precise estimates. Therefore, t-statistics should be larger. That’s a pretty simple idea. And that’s going to be the starting point of Card and Krueger’s paper. They’re gonna ask whether this actually holds. Not in a literature with like three studies.

Skip to 0 minutes and 40 seconds There’s gonna be 15 to 20 studies by the time they do their review in the mid-‘90s. So, a decent body of literature on a very important policy question.

Skip to 0 minutes and 51 seconds They’re gonna be focusing on what happens to employment when the federal minimum wage goes up. Critical policy question. And what they find is – Their starting point is to say, well, there’s this kind of older cross-sectional literature of these 15 to 20 studies. And they tend to find these negative employment effects. But the more recent cross-sectional literature, using better research designs actually, tend to find pretty minimal effects. Either small or negative effects, or zero effects. I said, okay, why is that? One possibility is the research design is different. You think you’re estimating the same thing, but you’re actually estimating something different. That’s one possibility. Another possibility they bring up, which is interesting as well, maybe effects are changing over time.

Skip to 1 minute and 36 seconds Like maybe back in the ‘70s, when the first studies came out, there was a certain effect of the minimum wage, but now in the ‘90s it’s different. Like the economy changed. Other policies changed. The labor market changed. There’s a couple of reasons why you might see differences in different sets of estimates. But their preferred explanation, what they’re gonna come to and show a lot of evidence for, isn’t that at all. They’re basically gonna argue that there’s just like pervasive publication bias in the older literature. And really, the evidence isn’t consistent with anything else. So, some combination of publication bias and specification searching is going to lead to a very misleading literature on an incredibly important policy question.

Skip to 2 minutes and 16 seconds Where leading scholars in the leading departments, are working on this for decades. And produced a couple dozen studies. Even in that case, they’re gonna say, “We really can’t trust this literature.” So that’s pretty important. This isn’t some esoteric sort of exercise. This is really important stuff. People are testifying in front of Congress about major economic policy changes. And based on 20 studies, they’re gonna say these studies do not have evidential value. Okay, so again, what we were just talking about, for a given effect size, and a given research design, if you double the sample size what’s gonna happen to precision? Precision is gonna go up by root two. Right? Straightforward. 41 percent.

Skip to 3 minutes and 0 seconds So sample size goes up given the same design. T-statistic should go up. So, do they see that? When they have more data, do they see t-statistics that are twice as large? Well, this is the square root of the degrees of freedom. This is the absolute t-ratio. There’s just this really striking pattern which is most of these studies have t-statistics just above two, in these cases. They’re just this cluster of studies right over here with values of two. And there’s a couple that are off that line. When they do regress the logged t-stat on logged degrees of freedom, instead of getting a slope of one, they get a negative slope. So this is deeply problematic.

Skip to 3 minutes and 49 seconds If it were zero, it would also be really problematic. It’s negative, so it’s like the folks out here, with big samples, given the same effect size – right? Big samples, should have bigger t-statistics. These guys out here just get zeroes. When you get lots of data, you get a zero. When you have a little bit of data, you just creep above a t of two somehow. You do a log specification rather than linear. You include four lags of past residential investment. You know, whatever. Whatever you need to do. That’s what you do. So this is a deeply troubling literature. It’s not one, it’s sort of negative one.

Skip to 4 minutes and 33 seconds Here’s another way of looking at the data.

Skip to 4 minutes and 37 seconds Again, we’ve got the standard error over here, and then this is the estimated effect. And what they plot out is two times the standard error. And you just get this incredible clustering of studies right above two times the standard error. Totally unnatural. And then again, what’s really concerning here is these studies with really small standard errors down here, these are the well powered studies, big degrees of freedom and whatnot, these are kind of like the small, the zero effects. Like the small effects. So, those are probably the most reliable studies.

Skip to 5 minutes and 19 seconds The bottom line is there’s this literature, 15 or 20 studies. Despite all that work, over a couple decades, it’s just not an informative literature. There’s so much just obvious, blatant publication bias in the literature, that we really don’t know what to make of it. And what they said is, “Look, there’s this recent cross-sectional study that’s well powered. With better designs, that shows zero effects. And that’s what we should be paying attention to.” And they’re like, “Oh, that’s our literature. Those are the papers we wrote.” And they were right.

"Time Series Minimum Wage Literature: A Meta-Analysis"

One way publication bias can have impacts beyond the scientific community is when poor policy recommendations are made based on skewed results. Such was the case when time-series minimum wage studies were used to make recommendations against raising the minimum wage in the United States. This video introduces David Card and Alan Krueger’s meta-analysis of these studies and demonstrates the effect of increasing an experiment’s sample size on the precision of its results.

In the article, economists David Card and Alan Krueger analyze the effect of an increase in the minimum wage on unemployment using aggregated time-series studies. Inspired by the widely-believed prediction that an increase in the minimum wage will lower the employment rates of low-wage workers, they estimated the probability of publication bias in studies on the relationship between changes the two variables.

Because more recent studies have found either smaller effects or marginally positive effects of the minimum wage on employment levels, and because of the role time-series evidence plays in minimum wage literature, Card and Krueger look to determine the validity of these claims.

They present a meta-analysis of published literature, building on the observation that “more recent studies have access to many more observations than earlier studies.” Card and Krueger define meta-analysis as “the quantitative analysis of a body of studies” that can be used to “summarize a set of related studies,” “evaluate the reliability of the findings in a statistical literature,” and “test for publication bias.”

They state that “basic sampling theory suggests that there should be a simple ‘inverse-square-root’ relationship between the sample size and the t ratio obtained in different studies.” However, their “findings are difficult to reconcile with the hypothesis that the literature contains an unbiased sample of the coefficients and t ratios that would be expected given the sample sizes used in the different studies.”

After finding t ratios that are actually negatively correlated with sample sizes, they conclude that “the time-series literature may have been affected by a combination of specification searching and publication bias, leading to a tendency for statistically significant results to be overrepresented in the published literature.” Note: specification searching is the technical term for what is widely known less formally as p-hacking.

Publication bias comes into play when authors are aware of reviewers’ tendency to give more credibility to studies with statistically significant results. Thus, economists who believe that a rise in minimum wage will lower employment may design their analyses (i.e., choose their variables, select their samples, specify their techniques, etc.) to generate the desired negative and significant effects.

In conclusion and considering reasons why published t ratios tend to equal 2 – the threshold significance value – regardless of the magnitude of the minimum-wage effect, Card and Krueger suggest two possible explanations:

  1. Structural change – While the true effects of changes in the minimum wage may have departed from earlier predictions, there is little incentive to report such changes because they challenge the validity of the existing time-series approach.

  2. Specification-searching and publication bias is a more plausible explanation, however – Because of the predominant theory, authors and editors tend to look for negative and statistically significant effects and will try to replicate and reproduce these results.

Due to the high probability of publication bias and specification searching, it is probable that “‘insignificant’ or ‘wrong-signed’ results may be substantially underreported in the published literature.”

You can read the full article here.


Card, David, and Alan B. Krueger. 1995. “Time-Series Minimum-Wage Studies: A Meta-Analysis.” The American Economic Review 85 (2): 238–43.

Share this video:

This video is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley