Skip main navigation

£199.99 £139.99 for one year of Unlimited learning. Offer ends on 28 February 2023 at 23:59 (UTC). T&Cs apply

Find out more

Methodology of Systematic Observational Research Process

So that’s for a single study. But now to solve that problem of publication bias and P-hacking. So the solution we came up with is we’ll stop doing one study at a time. Let’s answer many questions in a single study. And so we picked as a proof of concept,depression, we identified 17 treatments that are frequently used to treat depression and actually it’s interesting the guidelines are not very specific on which one you should use. It’s just a doctor should just pick one depending on patient preferences, doctor preferences and well… So there’s really a need for for evidence in this place. And we came up… we made all possible comparisons of these seventeen treatments.
So 17 times 17 minus in diagonal is 272 comparisons. We identified 22 outcomes of interest including stroke, insomnia, suicidal ideation, and some others. And we basically, within every comparison, we looked at all 22 outcomes, meaning that in total we had almost six thousand research questions that we want answers.
As I just showed you, you need different positive controls. So we identified negative controls, and we automatically generated positive controls from that. So we had 56,000 control questions.
And so we ran our our fancy machine with the large scale regularized regression on all of these research questions. That was quite a lot of computation. And we actually did this for four databases. And so in the end we had estimates for all of those, and we were able to do the empirical calibration and we came up with 6,000 calibrated estimates for every database that we use. So the end result looks like this. So every one of these dots is an empirically calibrated estimate for a comparison between two depression treatments for one of those 22 outcomes of interest.
Of course, this graph looks very different from the one that we saw in literature. And the reason is obvious there because there is no publication bias here. Nobody is saying you can’t show this dot because it’s not specific statistical significant and there’s no P hacking. So nobody’s moving the dots over the line just to get it statistically significant. So I would say that not only do we have informational small and no effect sizes, we also don’t have P hacking and publication bias. And so we actually, just last week, we published this paper where we we explain this process.
I really would invite you to go to this URL I’m gonna to leave it up for a couple of seconds. Because that has all the evidence that I just showed you, you can actually just browse through it. And click on every one of those points and see that each and every one of the estimates has the full information that you would expect. Like the covariate balance information, the information on the negative and positive controls etc. What’s… just figuring that out isn’t enough. We need a third step that’s actually now… now that we believe that we have a solution.
We need to actually do it and I think a lot of time methodological researchers like me fall into this trap that we have a solution. We publish our paper and then that’s it. But no! We need to actually start doing this and so that’s where the large-scale evidence generation and evaluation in a network of databases or legend for short project comes in.

Dr. Martijn Schuemie explains the methodology of solving that publication bias and P-hacking problem. They have made all possible comparisons of those seventeen treatments.

You can check the URL here: that he mentioned.

This article is from the free online

AI and Big Data in Global Health Improvement

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education