Skip to 0 minutes and 1 secondHow widely can we apply these ideas? The one sort of direct application that I think is quite straightforward is laboratory experiments. Experiments are prospective. They’re experiments. You have to basically have some sort of research design in mind when you design the study. So if you have a research design in mind, presumably you have a test in mind that you are going to run. I have done some experiments and I’ve actually enjoyed doing them. Recently I learned a lot from working with experimental economists. But the fear from the outside of experimental economists is that experimental economists are showing us 5 percent of all the tests that they’ve run. Now, I’m exaggerating it, but that’s the fear from the outside.
Skip to 0 minutes and 43 secondsBecause we don’t know what labs they’ve run. So this is a case where it seems like low hanging fruit.
Skip to 0 minutes and 52 secondsMuriel Niederle at Stanford and her co-author, Kauffman, take the opposite approach. Their point is like, “Look! Labs are cheap to run.” You can run a lab for a thousand dollars, two thousand dollars or something like that. So in that case, you don’t really need to pre-specify. Any weird, interesting finding that comes out of a lab should just be replicated by somebody else. And maybe replicated ten times. Okay, so the cheaper the cost of replication is – Like if it were literally costless, then maybe pre-specification isn’t as essential because at low cost, you can check if this actually holds. This is an interesting point that Muriel makes.
Skip to 1 minute and 28 secondsThat said, it would really be nice to know all the labs that are run. And it would be really nice if all this data were shared for the research community. And the only way we are going to know what was run, and the only way we are going to be able to get access to data and even know that the data exists, is if stuff is pre-registered. Prospective non-experimental studies. This is another category of studies. Prospective means, you know, “pre-planned.” Like, I planned to do something or collect data or get my hands on data before I have it. It’s like a natural experiment, or a policy experiment, or a quasi-experimental, or whatever term you want to use.
Skip to 2 minutes and 4 secondsAnd you still have a research design that is a prospective research design. So you could pre-register whenever new data is coming out that is not yet available. And I think we’ve alluded to this, so I’m not going to sort of dwell on it, but a new census round is coming out. A new round of the PSAID is coming out. A new round of a panel survey in Ghana is coming out. Before it’s released, you can write down what you would run and there is no way you could have mined that data. Registering observational studies. Let’s talk about Dal-Re et al. They have a much more sort of provocative position on registration of research.
Skip to 2 minutes and 42 secondsThe kind of starting point is, “Well, you know, we know we can register medical trials. We’ve been doing that for a long time and we’ve made progress there. We know we can do field trials.” Remember there are hundreds of these field trials getting registered. “We think it’s pretty straightforward to register labs experiments. And we also think, based on the discussions we just had, sure, there are some limitations, but it is kind of straightforward to register a prospective non-experimental study.” This is what I’m going to do with the data. You register that. What about the non-experimental, observational stuff that may not be prospective? What do we do with that work? How do we bring transparency practices into that realm?
Skip to 3 minutes and 18 secondsAnd, in particular, specifically, can we register these studies? The first thing they do is they did a search on all published medical papers in the year 2011. There were 400,000 papers.
Skip to 3 minutes and 33 secondsRCTs were 6 percent of studies. These are studies involving human data. So it’s not like animal studies, or DNA micro arrays or – it’s like human beings. 400,000 – 6 percent were RCTs. So again, for those of us from the outside, we think of medical research, we think of RCTs. No, over 90 percent were observational studies. That’s where the bulk of research is. What are we going to do with all this observational research? How can we bring transparency practices here? There are like five or six of these dueling journal editorial statements in the leading epidemiology journals about whether observational studies should be registered. And most of the journals came out and said, “No. We just don’t think it’s practical.”
Skip to 4 minutes and 13 secondsA couple of them came out in favor. And this Del-Re et al. comes out in favor. Why do Del-Re et al. make this case? They have a very strong call in favor. And they make some interesting arguments. So I want to talk about the arguments in favor. And then they also talk about the arguments against it. So it’s a nice piece because it kind of lays out the arguments and it ties into a lot of what we’re discussing. So part of what we – we were just mentioning this before – They’re like, “Hey, if you’re using human subjects’ data, you’re getting IRB approval. You’re already telling somebody what you’re planning to do with the data, so just post it.”
Skip to 4 minutes and 45 secondsPost it online. It’s low cost. You’re telling an ethics board how you are going to use human subjects’ data. What’s the big deal? It’s not burdensome.” That’s their first point. Second point, they want to make more evidence visible. If everything that got IRB approval got registered, then even papers that weren’t written up would be visible. Huge fractions of null results are never written up. So, like, hey, we don’t want that. You plan to do this analysis. Maybe you didn’t publish it because it was a null result. Maybe you didn’t publish it for some other reason. I should be able to search a registry or something.
Skip to 5 minutes and 19 secondsThere should be some public record that you plan to work on this or you were working on this. So I can try to contact you for your results, so I can do a meta-analysis. So I can combine results across studies. It sounds pretty good. Three, speculatively – They don’t call it speculative. I’m calling it speculative because I think it’s speculative. I just don’t think there’s evidence for this yet. They say, “You know what? More people are going to publish their results if there’s a registry.” It’s not implausible. Now there’s a record that you plan to work on something. You did the analysis. Maybe you got a null result. Maybe something else happened.
Skip to 5 minutes and 49 secondsBut people know you were planning to work on it. Maybe you’ll put in that extra two days of work and send it off to a journal. Even a low-ranking journal just to get it out there. And if no one knew about it, you’d move on to something else. But the way they put it, and they have a nice quote in the paper about this. I didn’t put it on the slide. You know, people will feel some sort of professional obligation to get work out there when they said they were going to work on it. What’s the downside? They talk about this downside. They kind of brush it off, but I think this is the big downside.
Skip to 6 minutes and 15 secondsAnd this is the point that comes up whenever there’s discussion that I’ve had with people about registering observational studies, which is the following. If we’re talking about registering studies using easily available, publicly available data, it may be impossible to verify whether the analysis preceded the registration. What’s to stop people who are dishonest? It gets back to the honesty point to some extent. But who are somewhat dishonest and who saw a related analysis in another paper – from basically registering something after they already know what the result is? We’re kind of on the honor system here. And the whole point of registration was to get away from the honor system.
How widely can we apply pre-registration?
Pre-registration is a great tool for prospective experimental studies, but how useful or feasible is it for observational studies, which make up the vast majority of social science research? In this video, Professor Miguel introduces two articles discussing this question and representing two very differing views. The first, written by Lucas Coffman and Muriel Niederle in 2015, presents an argument against using pre-analysis plans for laboratory experiments, while the second, written by Rafael Dal-Ré and colleagues in 2014, argues for the use of pre-analysis plans in observational studies.
If you have time, both articles mentioned in this paper are worth reading. We discuss them further below.
In the first article “Pre-analysis plans have limited upside, especially where replications are feasible”, economists Lucas Coffman and Muriel Niederle discuss the benefits and costs of using pre-analysis plans, study registries, and performing replications.
They argue that pre-analysis plans, while valuable for very large, unique studies, are less useful for studies in which multiple hypotheses are tested, when null results are likely to be unreported, or when replication is feasible and likely. They also argue that “recent empirical literature suggests the behavioral problems that pre-analysis plans attenuate are not a pervasive problem in experimental economics” and that “pre-analysis plans may discourage the use of novel research designs and hence inhibit studies of robustness of previous findings.”
They are especially concerned with how the requirement of a pre-analysis plan might discourage exploratory research: “…the costs for exploratory work may be increased relative to somewhat more derivative work as a researcher may be reluctant to head into uncharted territory if the researcher has to commit to a rigid pre-analysis plan beforehand.”
On the other hand, Rafael Dal-Ré, John Ioannidis, and six of their colleagues argue in “Making prospective registration of observational research a reality” that the registration of a pre-analysis plan can facilitate future meta-analyses and address the “file drawer problem” we discussed earlier:
“There are several postulated benefits in systematically registering all OSs [observational studies]: increasing transparency and credibility, improving the peer-review process and ethical conduct of studies, and ensuring that the totality of evidence is publicly available. Moreover, registration of OSs may enhance communication regarding explored, but not published, hypotheses, facilitate systematic reviews and research collaborations, and reduce redundancy and funding committed to research questions for which adequate studies have already been conducted or are being performed, allowing published evidence to be better placed in context.”
These authors also address Coffman and Niederle’s concerns that pre-analysis plans discourage exploratory research:
“A potential disadvantage of registering highly exploratory, hypothesis-generating research with complex, meandering analyses is the burden of ongoing serial amendments and the resulting hindrance of new idea generation, as well as reduction in the analyses of end points not prespecified because they were conceived after the study started. However, there is no compelling reason why new concepts should be hindered; they just need to be identified as post hoc observations. Such disclosure allows others to fully understand and openly debate the nature and merit of the analyses. There is no evidence that registering CTs [clinical trials] has led to fewer hypotheses being tested or a decline in secondary analysis of trial data. Conversely, there is greater recognition that hypotheses and analyses for testing them need to be specified a priori; without such delineation, study results can lead to biased reframing of the hypothesis or cherry-picking among unspecified end points.”
The authors call for journal editors, as well as research funders, to endorse registration in the same way that they have required Institutional Review Board (IRB) approval. Moreover, if IRB approval is required, a pre-analysis plan does not require much additional work.
What do you think? Do the benefits of using and registering a pre-analysis plan outweigh the costs of not doing so?
You can read the entirety of both papers by clicking on the links in the SEE ALSO section at the bottom of this page.
Coffman, Lucas C., and Muriel Niederle. 2015. “Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible.” Journal of Economic Perspectives 29 (3): 81–98.
Dal-Ré, Rafael, John P. Ioannidis, Michael B. Bracken, Patricia A. Buffler, An-Wen Chan, Eduardo L. Franco, Carlo La Vecchia, and Elisabete Weiderpass. 2014. “Making Prospective Registration of Observational Research a Reality.” Science Translational Medicine 6 (224): 224cm1-224cm1. doi:10.1126/scitranslmed.3007513.
© Center for Effective Global Action