Skip to 0 minutes and 1 secondCan the existence of a registry help us deal with hidden findings? The answer is going to be “sort of.” So, medicine is way ahead of the rest of us. was launched in ’99 and has become much more widely used in the last decade. What ended up happening is a whole bunch of journal editors got together in 2005 and said, “We’re basically not going to publish your clinical trial anymore unless you’ve pre-registered it.” The NIH also started really cracking down on this for funding. The FDA started cracking down. So there was this whole sort of squeeze about ten years ago in medical research.

Skip to 0 minutes and 37 secondsAnd what you can see is, there are lots of medical trials and you know, a bunch were getting registered starting in 2000. Once the policy changed for the journals, then it just shot up. So what does Mathieu et al do? They look at three specific fields in some leading journals and just see how closely the published papers – How consistent they are with the registration of those particular studies. They want to get at the extent of selection reporting and they find there is imperfect correspondence. So I think it’s worth talking just a little bit about how exactly that is the case. A couple of summary stats on these articles. There are about 300 articles in total that they focus on.

Skip to 1 minute and 16 secondsIn a particular time period, in particular sub-fields with published articles. A couple of things to note. The majority of these studies were funded by industry. This is a little bit of a different world than our social science world. Most published studies in economics, psychology, political science and sociology journals are not published by researchers sponsored by industry. That’s a big, big difference. And again, there is this concern that that is contaminating research. Of course, there is a lot of university funding and whatnot. They are kind of spread across fields. So what goes missing? What do they find? And I want to just focus on the sort of ones with the arrows.

Skip to 2 minutes and 4 secondsA large share of the studies focus on a different primary outcome in the text than they did in the registration. So in the medical registry it’s a little different than some of the social science registries. They really pin you down on what is the primary outcome. Is it this particular condition? Is it mortality? And then you have secondary outcomes and you have exploratory outcomes. So, you’re supposed to focus on your main outcome and you should always be reporting it somewhere. I mean, I think that’s the idea. Even if, for whatever reason, you emphasize other things. If you registered that you were going to look at mortality and then you suppress that, it feels like you are being a bit shifty.

Skip to 2 minutes and 44 secondsBut in 10 percent of cases, it was omitted in the text.

Skip to 2 minutes and 51 secondsIs that high or low? What’s your intuition or your feeling on that? Do you read this and say, “Oh, my God! I can’t trust medical research!”? I had a little bit of a different gut reaction. I was like, “Wow, these guys are better than us.”

Skip to 3 minutes and 6 secondsThe registry hasn’t solved all problems. There are still a lot of problems. But, at a minimum, now we can see them. I mean, this is like the great advantage of the registry. We can do this and we can argue over this number. Is this high or low? Or we can say, “Well, maybe half the time the data was missing.” We can really understand where the results disappear now. And before the medical registries existed, we had no way of doing this. We were like we are in social science now. We just really don’t even know what the literature is telling us half the time because we don’t know how many papers went unpublished.

Skip to 3 minutes and 36 secondsThe other thing down here was, “Are there discrepancies in the primary outcomes relative to the registration that favor significant outcomes?” Meaning, did they change their specification along the way or pick a robustness check to emphasize because it was more significant? Again, this is a little bit subjective. Half the time they say, “Well, we can’t really tell.” 10 percent of the time they say, “No”. But they say, “Around 40 percent of the time, we still think there is some selective reporting.” So maybe this number, together with this number, now we’re like, “Okay, there’s still quite...” I mean the bottom line of Mathieu et al is that there is still quite a bit of selective reporting despite the registry.

Skip to 4 minutes and 15 secondsSo there are different ways of viewing that. One way of viewing it is, “Well, gosh! This registry doesn’t work. There are all these problems. They have a registry that doesn’t work. We’re wasting our time with the registry. The other way to view it is sort of the point I was making before – Several of us were making. Which is like, “Now we can diagnose these problems. We can understand them. We can quantify them. We can work to improve them.” Like, all these articles – this article got published in JAMA. You know, millions of doctors read that article. These articles have an impact. And they keep pushing medicine to sort of have better and better rules around reporting.

Can the existence of a registry help us deal with hidden findings?

The medical sciences have led the way in requiring pre-registration, leading to substantial growth in their use. But how effective is the registry in shifting the reporting of actual results? In 2009, Sylvain Mathieu and his colleagues set out to answer this question by comparing collected data and results reported in medical journals. Their findings demonstrate ample room for improvement.

We will explore pre-registration and pre-analysis plans further next week.

Share this video:

This video is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley