How widely can we apply pre-registration?
In the first article “Pre-analysis plans have limited upside, especially where replications are feasible”, economists Lucas Coffman and Muriel Niederle discuss the benefits and costs of using pre-analysis plans, study registries, and performing replications.They argue that pre-analysis plans, while valuable for very large, unique studies, are less useful for studies in which multiple hypotheses are tested, when null results are likely to be unreported, or when replication is feasible and likely. They also argue that “recent empirical literature suggests the behavioral problems that pre-analysis plans attenuate are not a pervasive problem in experimental economics” and that “pre-analysis plans may discourage the use of novel research designs and hence inhibit studies of robustness of previous findings.”They are especially concerned with how the requirement of a pre-analysis plan might discourage exploratory research: “…the costs for exploratory work may be increased relative to somewhat more derivative work as a researcher may be reluctant to head into uncharted territory if the researcher has to commit to a rigid pre-analysis plan beforehand.”On the other hand, Rafael Dal-Ré, John Ioannidis, and six of their colleagues argue in “Making prospective registration of observational research a reality” that the registration of a pre-analysis plan can facilitate future meta-analyses and address the “file drawer problem” we discussed earlier:
“There are several postulated benefits in systematically registering all OSs [observational studies]: increasing transparency and credibility, improving the peer-review process and ethical conduct of studies, and ensuring that the totality of evidence is publicly available. Moreover, registration of OSs may enhance communication regarding explored, but not published, hypotheses, facilitate systematic reviews and research collaborations, and reduce redundancy and funding committed to research questions for which adequate studies have already been conducted or are being performed, allowing published evidence to be better placed in context.”
The authors call for journal editors, as well as research funders, to endorse registration in the same way that they have required Institutional Review Board (IRB) approval. Moreover, if IRB approval is required, a pre-analysis plan does not require much additional work.“A potential disadvantage of registering highly exploratory, hypothesis-generating research with complex, meandering analyses is the burden of ongoing serial amendments and the resulting hindrance of new idea generation, as well as reduction in the analyses of end points not prespecified because they were conceived after the study started. However, there is no compelling reason why new concepts should be hindered; they just need to be identified as post hoc observations. Such disclosure allows others to fully understand and openly debate the nature and merit of the analyses. There is no evidence that registering CTs [clinical trials] has led to fewer hypotheses being tested or a decline in secondary analysis of trial data. Conversely, there is greater recognition that hypotheses and analyses for testing them need to be specified a priori; without such delineation, study results can lead to biased reframing of the hypothesis or cherry-picking among unspecified end points.”
Want to keep
University of California, Berkeley online course,
Transparent and Open Social Science Research
ReferencesCoffman, Lucas C., and Muriel Niederle. 2015. “Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible.” Journal of Economic Perspectives 29 (3): 81–98.Dal-Ré, Rafael, John P. Ioannidis, Michael B. Bracken, Patricia A. Buffler, An-Wen Chan, Eduardo L. Franco, Carlo La Vecchia, and Elisabete Weiderpass. 2014. “Making Prospective Registration of Observational Research a Reality.” Science Translational Medicine 6 (224): 224cm1-224cm1. doi:10.1126/scitranslmed.3007513.
Transparent and Open Social Science Research
Our purpose is to transform access to education.
We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.
We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.