IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0003081.html
   My bibliography  Save this article

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

Author

Listed:
  • Kerry Dwan
  • Douglas G Altman
  • Juan A Arnaiz
  • Jill Bloom
  • An-Wen Chan
  • Eugenia Cronin
  • Evelyne Decullier
  • Philippa J Easterbrook
  • Erik Von Elm
  • Carrol Gamble
  • Davina Ghersi
  • John P A Ioannidis
  • John Simes
  • Paula R Williamson

Abstract

Background: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings: We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions: Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Suggested Citation

  • Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
  • Handle: RePEc:plo:pone00:0003081
    DOI: 10.1371/journal.pone.0003081
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0003081
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0003081&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0003081?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. J. L. Hutton & Paula R. Williamson, 2000. "Bias in meta‐analysis due to outcome variable selection within studies," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 49(3), pages 359-370.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nikolaos Pandis & Padhraig S Fleming & Helen Worthington & Kerry Dwan & Georgia Salanti, 2015. "Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
    2. Wynanda A van Enst & Rob J P M Scholten & Lotty Hooft, 2012. "Identification of Additional Trials in Prospective Trial Registers for Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 7(8), pages 1-5, August.
    3. Jamie J Kirkham & Doug G Altman & Paula R Williamson, 2010. "Bias Due to Changes in Specified Outcomes during the Systematic Review Process," PLOS ONE, Public Library of Science, vol. 5(3), pages 1-5, March.
    4. Piia K. Peura & Janne A. Martikainen & Timo T. Purmonen & Juha H. O. Turunen, 2012. "Sponsorship-Related Outcome Selection Bias in Published Economic Studies of Triptans," Medical Decision Making, , vol. 32(2), pages 237-245, March.
    5. D. J. Bartholomew, 2002. "Discussion on the paper by Fayers and Hand," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(2), pages 253-261, June.
    6. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    7. Peter M. Fayers & David J. Hand, 2002. "Causal variables, indicator variables and measurement scales: an example from quality of life," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(2), pages 233-253, June.
    8. Jing Wang & Qiguo Zhang & Rongfu Zhou & Bing Chen & Jian Ouyang, 2012. "High-Dose Chemotherapy Followed by Autologous Stem Cell Transplantation as a First-Line Therapy for High-Risk Primary Breast Cancer: A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 7(3), pages 1-8, March.
    9. Peter Boone & Alex Eble & Diana Elbourne, 2013. "Risk and Evidence of Bias in Randomized Controlled Trials in Economics," CEP Discussion Papers dp1240, Centre for Economic Performance, LSE.
    10. Dan Jackson & John Copas & Alex J. Sutton, 2005. "Modelling reporting bias: the operative mortality rate for ruptured abdominal aortic aneurysm repair," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(4), pages 737-752, November.
    11. Salandra, Rossella & Criscuolo, Paola & Salter, Ammon, 2021. "Directing scientists away from potentially biased publications: the role of systematic reviews in health care," Research Policy, Elsevier, vol. 50(1).
    12. Salandra, Rossella, 2018. "Knowledge dissemination in clinical trials: Exploring influences of institutional support and type of innovation on selective reporting," Research Policy, Elsevier, vol. 47(7), pages 1215-1228.
    13. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," The World Bank Economic Review, World Bank, vol. 31(3), pages 687-707.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0003081. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.