Publication bias (Part 1)

Publication bias is a type of reporting bias, specifically the selective publication of studies with statistically significant outcomes.

Published studies may not represent the entire population of completed studies. Reviewers might draw incorrect conclusions, leading to over-optimistic results or the false perception of an ineffective or dangerous treatment as safe and effective.

Several factors contribute to publication bias, such as the tendency of journals to reject negative studies, the preference for statistically significant results, small sample sizes, researchers deciding not to submit, and the execution of reviews and meta-analyses.

Publication bias can lead to several issues, including Type-I errors, overestimation of effect size, and questionable research practices. Publication bias can be assessed visually using a funnel plot.

Funnel plot

A funnel plot is a scatter plot of individual studies, with dots representing each study. It helps visualise the presence of publication bias. In a funnel plot with no bias, the dots are symmetrically distributed around the vertical line, indicating no publication bias. In a funnel plot with bias, the dots are asymmetrically distributed, indicating the presence of publication bias.

Here, we compare funnel plots with and without publication bias. Notice the difference in the distribution of dots.

Sometimes, funnel plots are rotated 90 degrees. In this funnel plot, the effect is plotted against the standard error. This plot shows meta-analysis weight against effect size, with 50 simulated trials and a true effect of 0.5. The boundaries are now straight lines, indicating no bias.

If only significant trials are published, parts of the funnel will be sparse or empty. This plot shows the effect against standard error, with clear diamonds representing trials where the difference is not significant.

What should you do if you have publication bias?

If you find possible evidence of publication bias, reconsider your search strategy, identify plausible explanations, explore potential explanations with sensitivity analyses, acknowledge all potential biases, and consider waiting for more studies to redo the systematic review.

There are several methods to deal with publication bias, including trim and fill, and selection models.

The trim-and-fill method

The trim and fill method is a popular way to correct effect size estimates by trimming missing effect sizes from one side of the funnel plot and filling them on the other side.

The trim-and-fill method is a two-step approach based on funnel plots designed to detect publication bias and adjust the results accordingly.

Phase 1 (Trimming): Small studies are excluded to create a symmetrical funnel plot, and an adjusted summary effect is estimated using only the larger studies.

Phase 2 (Filling): The funnel plot is reconstructed by adding ‘missing’ counterparts of the excluded studies around the adjusted summary estimate.

Selection models

Selection models focus on the process by which trials are chosen for publication, emphasising the mechanism behind study selection. These models allow researchers to estimate the potential impact of missing studies if they had been included in the meta-analysis.

A key assumption of selection models is that the included studies are not randomly selected; they are published due to specific characteristics that increase their likelihood of publication.

Consequently, the overall estimate is conditional on the studies that have been published and identified. By accounting for this, researchers can calculate the marginal effect size, which represents the effect size independent of publication status.

References:

• Dimitris Mavridis, Georgia Salanti – How to assess publication bias: funnel plot, trim-and-fill method and selection models: Evidence-Based Mental Health 2014;17.

• Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics. 2018 Sep;74(3):785-794. doi: 10.1111/biom.12817. Epub 2017 Nov 15. PMID: 29141096; PMCID: PMC5953768.

• Joober R, Schmitz N, Annable L, Boksa P. Publication bias: what are the challenges and can they be overcome? J Psychiatry Neurosci. 2012 May;37(3):149-52. doi: 10.1503/jpn.120065. PMID: 22515987; PMCID: PMC3341407.