Want to keep learning?

This content is taken from the University of California, Berkeley, Center for Effective Global Action (CEGA) & Berkeley Initiative for Transparency in the Social Sciences (BITSS)'s online course, Transparent and Open Social Science Research. Join the course to learn more.

Skip to 0 minutes and 1 secondHow widely can we apply these ideas? The one sort of direct application that I think is quite straightforward is laboratory experiments. Experiments are prospective. They’re experiments. You have to basically have some sort of research design in mind when you design the study. So if you have a research design in mind, presumably you have a test in mind that you are going to run. I have done some experiments and I’ve actually enjoyed doing them. Recently I learned a lot from working with experimental economists. But the fear from the outside of experimental economists is that experimental economists are showing us 5 percent of all the tests that they’ve run. Now, I’m exaggerating it, but that’s the fear from the outside.

Skip to 0 minutes and 43 secondsBecause we don’t know what labs they’ve run. So this is a case where it seems like low hanging fruit.

Skip to 0 minutes and 52 secondsMuriel Niederle at Stanford and her co-author, Kauffman, take the opposite approach. Their point is like, “Look! Labs are cheap to run.” You can run a lab for a thousand dollars, two thousand dollars or something like that. So in that case, you don’t really need to pre-specify. Any weird, interesting finding that comes out of a lab should just be replicated by somebody else. And maybe replicated ten times. Okay, so the cheaper the cost of replication is – Like if it were literally costless, then maybe pre-specification isn’t as essential because at low cost, you can check if this actually holds. This is an interesting point that Muriel makes.

Skip to 1 minute and 28 secondsThat said, it would really be nice to know all the labs that are run. And it would be really nice if all this data were shared for the research community. And the only way we are going to know what was run, and the only way we are going to be able to get access to data and even know that the data exists, is if stuff is pre-registered. Prospective non-experimental studies. This is another category of studies. Prospective means, you know, “pre-planned.” Like, I planned to do something or collect data or get my hands on data before I have it. It’s like a natural experiment, or a policy experiment, or a quasi-experimental, or whatever term you want to use.

Skip to 2 minutes and 4 secondsAnd you still have a research design that is a prospective research design. So you could pre-register whenever new data is coming out that is not yet available. And I think we’ve alluded to this, so I’m not going to sort of dwell on it, but a new census round is coming out. A new round of the PSAID is coming out. A new round of a panel survey in Ghana is coming out. Before it’s released, you can write down what you would run and there is no way you could have mined that data. Registering observational studies. Let’s talk about Dal-Re et al. They have a much more sort of provocative position on registration of research.

Skip to 2 minutes and 42 secondsThe kind of starting point is, “Well, you know, we know we can register medical trials. We’ve been doing that for a long time and we’ve made progress there. We know we can do field trials.” Remember there are hundreds of these field trials getting registered. “We think it’s pretty straightforward to register labs experiments. And we also think, based on the discussions we just had, sure, there are some limitations, but it is kind of straightforward to register a prospective non-experimental study.” This is what I’m going to do with the data. You register that. What about the non-experimental, observational stuff that may not be prospective? What do we do with that work? How do we bring transparency practices into that realm?

Skip to 3 minutes and 18 secondsAnd, in particular, specifically, can we register these studies? The first thing they do is they did a search on all published medical papers in the year 2011. There were 400,000 papers.

Skip to 3 minutes and 33 secondsRCTs were 6 percent of studies. These are studies involving human data. So it’s not like animal studies, or DNA micro arrays or – it’s like human beings. 400,000 – 6 percent were RCTs. So again, for those of us from the outside, we think of medical research, we think of RCTs. No, over 90 percent were observational studies. That’s where the bulk of research is. What are we going to do with all this observational research? How can we bring transparency practices here? There are like five or six of these dueling journal editorial statements in the leading epidemiology journals about whether observational studies should be registered. And most of the journals came out and said, “No. We just don’t think it’s practical.”

Skip to 4 minutes and 13 secondsA couple of them came out in favor. And this Del-Re et al. comes out in favor. Why do Del-Re et al. make this case? They have a very strong call in favor. And they make some interesting arguments. So I want to talk about the arguments in favor. And then they also talk about the arguments against it. So it’s a nice piece because it kind of lays out the arguments and it ties into a lot of what we’re discussing. So part of what we – we were just mentioning this before – They’re like, “Hey, if you’re using human subjects’ data, you’re getting IRB approval. You’re already telling somebody what you’re planning to do with the data, so just post it.”

Skip to 4 minutes and 45 secondsPost it online. It’s low cost. You’re telling an ethics board how you are going to use human subjects’ data. What’s the big deal? It’s not burdensome.” That’s their first point. Second point, they want to make more evidence visible. If everything that got IRB approval got registered, then even papers that weren’t written up would be visible. Huge fractions of null results are never written up. So, like, hey, we don’t want that. You plan to do this analysis. Maybe you didn’t publish it because it was a null result. Maybe you didn’t publish it for some other reason. I should be able to search a registry or something.

Skip to 5 minutes and 19 secondsThere should be some public record that you plan to work on this or you were working on this. So I can try to contact you for your results, so I can do a meta-analysis. So I can combine results across studies. It sounds pretty good. Three, speculatively – They don’t call it speculative. I’m calling it speculative because I think it’s speculative. I just don’t think there’s evidence for this yet. They say, “You know what? More people are going to publish their results if there’s a registry.” It’s not implausible. Now there’s a record that you plan to work on something. You did the analysis. Maybe you got a null result. Maybe something else happened.

Skip to 5 minutes and 49 secondsBut people know you were planning to work on it. Maybe you’ll put in that extra two days of work and send it off to a journal. Even a low-ranking journal just to get it out there. And if no one knew about it, you’d move on to something else. But the way they put it, and they have a nice quote in the paper about this. I didn’t put it on the slide. You know, people will feel some sort of professional obligation to get work out there when they said they were going to work on it. What’s the downside? They talk about this downside. They kind of brush it off, but I think this is the big downside.

Skip to 6 minutes and 15 secondsAnd this is the point that comes up whenever there’s discussion that I’ve had with people about registering observational studies, which is the following. If we’re talking about registering studies using easily available, publicly available data, it may be impossible to verify whether the analysis preceded the registration. What’s to stop people who are dishonest? It gets back to the honesty point to some extent. But who are somewhat dishonest and who saw a related analysis in another paper – from basically registering something after they already know what the result is? We’re kind of on the honor system here. And the whole point of registration was to get away from the honor system.

How widely can we apply pre-registration?

Pre-registration is a great tool for prospective experimental studies, but how useful or feasible is it for observational studies, which make up the vast majority of social science research? In this video, Professor Miguel introduces two articles discussing this question and representing two very differing views. The first, written by Lucas Coffman and Muriel Niederle in 2015, presents an argument against using pre-analysis plans for laboratory experiments, while the second, written by Rafael Dal-Ré and colleagues in 2014, argues for the use of pre-analysis plans in observational studies.

Share this video:

This video is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley