A paper on publication bias in time-series minimum wage literature by Card and Krueger
In this article, economists David Card and Alan Krueger analyze the effect of an increase in the minimum wage on unemployment using aggregated time-series studies. They estimate the probability of publication bias in studies on the relationship between these two variables (change in minimum wage and level of employment), inspired by the widely-known prediction that an increase in the minimum wage will lower the employment rates of low-wage workers.
Because more recent studies have found either smaller effects or marginally positive effects of the minimum wage on employment levels, and because of the role time-series evidence plays in minimum-wage literature, Card and Krueger look to validate the time-series estimates.
They present a meta-analysis of previously published literature, building on the observation that “more recent studies have access to many more observations than earlier studies.” Card and Krueger define meta-analysis as “the quantitative analysis of a body of studies” that can be used to “summarize a set of related studies,” “evaluate the reliability of the findings in a statistical literature,” and “test for publication bias.”
In their analysis, they state that “basic sampling theory suggests that there should be a simple ‘inverse-square-root’ relationship between the sample size and the t ratio obtained in different studies.” However, their “findings are difficult to reconcile with the hypothesis that the literature contains an unbiased sample of the coefficients and t ratios that would be expected given the sample sizes used in the different studies.”
After finding t ratios that are actually negatively correlated with sample sizes, they conclude that “the time-series literature may have been affected by a combination of specification searching and publication bias, leading to a tendency for statistically significant results to be overrepresented in the published literature.” Note: specification searching is the technical term for what is also widely known as p-hacking.
Publication bias comes into play when authors are aware of reviewers’ tendency to give more credibility to studies with statistically significant results. Thus, economists, with their belief that a rise in minimum wage will lower employment, may design their analysis (i.e. choose their variables, select their samples, specify their techniques, etc.) to generate the desired negative and significant effects.
In conclusion and considering reasons why t ratios tend to equal 2 – the threshold significance value – regardless of the magnitude of the minimum-wage effect, Card and Krueger suggest two possible explanations:
Structural change – While the true effects of changes in the minimum wage may have departed from earlier predictions, there is little incentive to report such changes because they challenge the validity of the existing time-series approach.
More plausible is specification-searching and publication bias – Because of the predominant theory, authors and editors tend to look for negative and statistically significant effects and will try to replicate and reproduce these results.
Due to the high probability of publication bias and specification searching, it is probable that “‘insignificant’ or ‘wrong-signed’ results may be substantially underreported in the published literature.”
Read the full article here.
Card, David, and Alan B. Krueger. 1995. “Time-Series Minimum-Wage Studies: A Meta-Analysis.” The American Economic Review 85 (2): 238–43.
© Center for Effective Global Action