Skip to 0 minutes and 1 secondCan the existence of a registry help us deal with hidden findings? The answer is going to be “sort of.” So, medicine is way ahead of the rest of us. Clinicaltrials.gov was launched in ’99 and has become much more widely used in the last decade. What ended up happening is a whole bunch of journal editors got together in 2005 and said, “We’re basically not going to publish your clinical trial anymore unless you’ve pre-registered it.” The NIH also started really cracking down on this for funding. The FDA started cracking down. So there was this whole sort of squeeze about ten years ago in medical research.

Skip to 0 minutes and 37 secondsAnd what you can see is, there are lots of medical trials and you know, a bunch were getting registered starting in 2000. Once the policy changed for the journals, then it just shot up. So what does Mathieu et al do? They look at three specific fields in some leading journals and just see how closely the published papers – How consistent they are with the registration of those particular studies. They want to get at the extent of selection reporting and they find there is imperfect correspondence. So I think it’s worth talking just a little bit about how exactly that is the case. A couple of summary stats on these articles. There are about 300 articles in total that they focus on.

Skip to 1 minute and 16 secondsIn a particular time period, in particular sub-fields with published articles. A couple of things to note. The majority of these studies were funded by industry. This is a little bit of a different world than our social science world. Most published studies in economics, psychology, political science and sociology journals are not published by researchers sponsored by industry. That’s a big, big difference. And again, there is this concern that that is contaminating research. Of course, there is a lot of university funding and whatnot. They are kind of spread across fields. So what goes missing? What do they find? And I want to just focus on the sort of ones with the arrows.

Skip to 2 minutes and 4 secondsA large share of the studies focus on a different primary outcome in the text than they did in the registration. So in the medical registry it’s a little different than some of the social science registries. They really pin you down on what is the primary outcome. Is it this particular condition? Is it mortality? And then you have secondary outcomes and you have exploratory outcomes. So, you’re supposed to focus on your main outcome and you should always be reporting it somewhere. I mean, I think that’s the idea. Even if, for whatever reason, you emphasize other things. If you registered that you were going to look at mortality and then you suppress that, it feels like you are being a bit shifty.

Skip to 2 minutes and 44 secondsBut in 10 percent of cases, it was omitted in the text.

Skip to 2 minutes and 51 secondsIs that high or low? What’s your intuition or your feeling on that? Do you read this and say, “Oh, my God! I can’t trust medical research!”? I had a little bit of a different gut reaction. I was like, “Wow, these guys are better than us.”

Skip to 3 minutes and 6 secondsThe registry hasn’t solved all problems. There are still a lot of problems. But, at a minimum, now we can see them. I mean, this is like the great advantage of the registry. We can do this and we can argue over this number. Is this high or low? Or we can say, “Well, maybe half the time the data was missing.” We can really understand where the results disappear now. And before the medical registries existed, we had no way of doing this. We were like we are in social science now. We just really don’t even know what the literature is telling us half the time because we don’t know how many papers went unpublished.

Skip to 3 minutes and 36 secondsThe other thing down here was, “Are there discrepancies in the primary outcomes relative to the registration that favor significant outcomes?” Meaning, did they change their specification along the way or pick a robustness check to emphasize because it was more significant? Again, this is a little bit subjective. Half the time they say, “Well, we can’t really tell.” 10 percent of the time they say, “No”. But they say, “Around 40 percent of the time, we still think there is some selective reporting.” So maybe this number, together with this number, now we’re like, “Okay, there’s still quite...” I mean the bottom line of Mathieu et al is that there is still quite a bit of selective reporting despite the registry.

Skip to 4 minutes and 15 secondsSo there are different ways of viewing that. One way of viewing it is, “Well, gosh! This registry doesn’t work. There are all these problems. They have a registry that doesn’t work. We’re wasting our time with the registry. The other way to view it is sort of the point I was making before – Several of us were making. Which is like, “Now we can diagnose these problems. We can understand them. We can quantify them. We can work to improve them.” Like, all these articles – this article got published in JAMA. You know, millions of doctors read that article. These articles have an impact. And they keep pushing medicine to sort of have better and better rules around reporting.

Can the existence of a registry help us deal with hidden findings?

The medical sciences have led the way in requiring pre-registration, leading to substantial growth in their use. But how effective is the registry in shifting the reporting of actual results? In 2009, Sylvain Mathieu and his colleagues set out to answer this question by comparing collected data and results reported in medical journals. Their findings demonstrate ample room for improvement.

We’ll explore pre-registration and pre-analysis plans further next week.


Registries were developed to address the “file drawer problem” and encourage more complete reporting of results. But because they are a relatively recent platform – the American Economics Association (AEA) RCT registry was recently established in 2012 – Sylvain Mathieu and four of his colleagues in the medical sciences decided to examine the impact of registries on reporting.

In 2005, the International Committee of Medical Journal Editors (ICMJE) began requiring the registration of clinical trials. As a precondition for publication in a journal, investigators were required to disclose information about their trial before enrolling participants.

In order to assess the prevalence of primary outcome reporting bias, the authors selected and examined 323 cardiology, rheumatology, and gastroenterology RCTs published in medical and specialty journals. Their main objective was “to help achieve transparency in results and to make information about the existence and design of clinical trials publicly available,” with the prediction that knowledge of these trials would reduce the risk of selective reporting.

Their main questions were:

  1. How many registered studies were eventually published in high impact journals?
  2. Of those studies, how many pre-registered primary outcomes were reported in the published articles?
  3. What was the proportion of statistically significant reported primary outcomes?

They also sought to compare general medical journals and specialty journals in terms of the proportion of registered trials and differences in primary outcomes.

Using the MEDLINE online medical database for studies and RCTs, the authors selected studies from high impact journals and for each study, established the funding sources, the number of reported outcomes, and their registration status.

Mathieu and his colleagues found that, while nearly half of the selected trials were adequately registered, 27.6% of trials had never been registered, 14% were registered only after the study was complete, and 12% were registered without any clear description of the study’s primary outcomes. They found no significant differences in reporting between general and specialty journals, though there were more adequately registered trials within general medical journals.

So does this mean that registries aren’t as useful as we may have hoped? Consider the AEA RCT registry. Though it was established less than five years ago, there are already over 1,000 registered trials on it – trials that can be accessed by readers looking to make more informed decisions regarding a study’s credibility or researchers attempting to accurately replicate a study.

If executed correctly, trial registration can improve accountability in planning and reporting, as well as safeguard against publication bias. Mathieu and his colleagues believe that the scientific community should endorse and adhere to these requirements, being cognizant of deadlines and using clarity when instructing authors on submission guidelines. They suggest quality control procedures and conclude that, with careful implementation, trial registration may ensure more robust results and unbiased reporting.

How can we inform more people of the significance of registries? Why might people be hesitant to register their studies?

You can read the whole paper here.


Reference

Mathieu S, Boutron I, Moher D, Altman DG, and Ravaud P. 2009. “Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials.” JAMA 302 (9): 977–84. doi:10.1001/jama.2009.1242.

Share this video:

This video is from the free online course:

Transparent and Open Social Science Research

University of California, Berkeley