"False-positive Psychology" by Simmons, Nelson, and Simonsohn

False-positives in social science research can have particularly negative impacts when research is used to inform policy or other outcomes. These outcomes can range, from the endorsement of the idea that voters’ attitudes about same-sex marriage can be changed by just a short conversation with a gay canvasser, to the suggestion that increasing austerity measures leads to economic growth.

In this article, authors Joseph Simmons, Leif Nelson, and Uri Simonsohn, address the costly issues and frequency of false positives in social science research and suggest six requirements for authors that they believe are simple, low-cost, and straight-forward solutions to the problem.

They first define a false positive as the “incorrect rejection of a null hypothesis”. The problem with false positives is that once they are published, they are persistent and there is little incentive to replicate findings to test their significance. The authors write: “..false positives waste resources: they inspire investment in fruitless research programs and can lead to ineffective policy changes. Finally, a field known for publishing false positives risks losing its credibility.”

Simmons, Nelson, and Simonsohn argue that, with the current standards of disclosure, false positives are, in fact, “vastly more likely”, due to the ease at which researchers can publish “statistically significant” evidence for any hypothesis.

Many of these issues can be attributed to researcher degrees of freedom, or the multitude of decisions researchers can make regarding the design and details of their experiments. Such freedom increases the likelihood that analyses will produce false positives, and can be attributed to two factors: “ambiguity in how to best make decisions, and the researcher’s desire to find a statistically significant result”, the latter factor referring to the temptation and likelihood of researchers coming to conclusions that are consistent with their own desires or beliefs.

The authors performed computer simulations to estimate the influence of researcher degrees of freedom on the probability of a false-positive result. They focused on four common degrees of freedom that increase the likelihood of a researcher falsely detecting a significant effect:

  1. Flexibility in choosing among dependent variables
  2. choosing sample size
  3. using covariates, and
  4. reporting subsets of experimental conditions.

They showed that if an effect is significant with a small sample size, then it would be significant with a larger one, too.

They also suggest six requirements as a solution to the high incidence false-positives, encouraging appropriate conduct of research, transparency in methods, and holding readers accountable to make informed decisions about the credibility of findings.

Authors must:

  1. decide the rule for terminating data collection before data collection begins and reports this rule in the article – This prevents authors from adding additional observations and further testing for significance if initial results are insignificant.

  2. collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification – Small samples are “simply not powerful enough to detect most effects” and are “more likely to reflect interim data analysis and a flexible termination rule”.

  3. list all variables collected in a study – This prevents researchers from only reporting convenient subsets of measurements and allows readers to identify degrees of freedom.

  4. report all experimental conditions, including failed manipulations – This “prevents authors from selectively choosing only to report conditions that yield results consistent with their hypothesis.”

  5. report the statistical results of eliminated observations if those observations had been included – This requires authors to explain why they eliminated the data and encourages readers to consider the validity of the data exclusion.

  6. report the statistical results of the analysis without the covariate if the analysis includes a covariate – This requires authors to “justify use of the covariate,” reveals “the extent to which a finding is reliant on the presence of a covariate,” and, again, encourages readers to practice discernment in whether the covariate is warranted.

Finally, Simmons, Nelson, and Simonsohn present four guidelines for reviewers to abide by.

Reviewers should…

  1. ensure that authors follow the requirements;
  2. be more tolerant of imperfections in results (false-positive findings could be due to an “unreasonable expectation” imposed by reviewers for data to turn out as predicted);
  3. require authors to demonstrate that their results do not hinge on arbitrary analytic decisions; and
  4. require the authors to conduct an exact replication if justifications of data collection or analysis are not compelling.

While there has been some criticism of the authors’ proposed solution, they still conclude that the requirements advocated in this article impose minimal costs to all involved in research and review and are a step towards the goal of discovering and disseminating valid research.

Think about the various researcher degrees of freedom for an experiment. Can you think of a time when degrees of freedom might’ve affected your own research?

You can read the entire article here: http://journals.sagepub.com/doi/pdf/10.1177/0956797611417632.


Reference

Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22 (11): 1359–66. doi:10.1177/0956797611417632.

Share this article:

This article is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley