IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0172650.html
   My bibliography  Save this article

Poor replication validity of biomedical association studies reported by newspapers

Author

Listed:
  • Estelle Dumas-Mallet
  • Andy Smith
  • Thomas Boraud
  • François Gonon

Abstract

Objective: To investigate the replication validity of biomedical association studies covered by newspapers. Methods: We used a database of 4723 primary studies included in 306 meta-analysis articles. These studies associated a risk factor with a disease in three biomedical domains, psychiatry, neurology and four somatic diseases. They were classified into a lifestyle category (e.g. smoking) and a non-lifestyle category (e.g. genetic risk). Using the database Dow Jones Factiva, we investigated the newspaper coverage of each study. Their replication validity was assessed using a comparison with their corresponding meta-analyses. Results: Among the 5029 articles of our database, 156 primary studies (of which 63 were lifestyle studies) and 5 meta-analysis articles were reported in 1561 newspaper articles. The percentage of covered studies and the number of newspaper articles per study strongly increased with the impact factor of the journal that published each scientific study. Newspapers almost equally covered initial (5/39 12.8%) and subsequent (58/600 9.7%) lifestyle studies. In contrast, initial non-lifestyle studies were covered more often (48/366 13.1%) than subsequent ones (45/3718 1.2%). Newspapers never covered initial studies reporting null findings and rarely reported subsequent null observations. Only 48.7% of the 156 studies reported by newspapers were confirmed by the corresponding meta-analyses. Initial non-lifestyle studies were less often confirmed (16/48) than subsequent ones (29/45) and than lifestyle studies (31/63). Psychiatric studies covered by newspapers were less often confirmed (10/38) than the neurological (26/41) or somatic (40/77) ones. This is correlated to an even larger coverage of initial studies in psychiatry. Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim. Conclusion: Journalists preferentially cover initial findings although they are often contradicted by meta-analyses and rarely inform the public when they are disconfirmed.

Suggested Citation

  • Estelle Dumas-Mallet & Andy Smith & Thomas Boraud & François Gonon, 2017. "Poor replication validity of biomedical association studies reported by newspapers," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-15, February.
    • Handle: RePEc:plo:pone00:0172650
    DOI: 10.1371/journal.pone.0172650
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172650
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0172650&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0172650?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Susan Watts, 2014. "Society needs more than wonder to respect science," Nature, Nature, vol. 508(7495), pages 151-151, April.
    2. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    3. Daniel Sarewitz, 2012. "Beware the creeping cracks of bias," Nature, Nature, vol. 485(7397), pages 149-149, May.
    4. Estelle Dumas-Mallet & Katherine Button & Thomas Boraud & Marcus Munafo & François Gonon, 2016. "Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-20, June.
    5. Leonard P Freedman & Iain M Cockburn & Timothy S Simcoe, 2015. "The Economics of Reproducibility in Preclinical Research," PLOS Biology, Public Library of Science, vol. 13(6), pages 1-9, June.
    6. François Gonon & Jan-Pieter Konsman & David Cohen & Thomas Boraud, 2012. "Why Most Biomedical Findings Echoed by Newspapers Turn Out to be False: The Case of Attention Deficit Hyperactivity Disorder," PLOS ONE, Public Library of Science, vol. 7(9), pages 1-11, September.
    7. Amélie Yavchitz & Isabelle Boutron & Aida Bafeta & Ibrahim Marroun & Pierre Charles & Jean Mantz & Philippe Ravaud, 2012. "Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study," PLOS Medicine, Public Library of Science, vol. 9(9), pages 1-11, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Briony Swire-Thompson & David Lazer, 2022. "Reducing Health Misinformation in Science: A Call to Arms," The ANNALS of the American Academy of Political and Social Science, , vol. 700(1), pages 124-135, March.
    2. Markus Lehmkuhl & Nikolai Promies, 2020. "Frequency distribution of journalistic attention for scientific studies and scientific sources: An input–output analysis," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-20, November.
    3. Peter Kardos & Ádám Kun & Csaba Pléh & Ferenc Jordán, 2023. "(How) should researchers publicize their research papers before peer review?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 2019-2023, March.
    4. Green, David A. & Simard-Duplain, Gaëlle & Sweetman, Arthur & Warburton, William P., 2023. "A Scientific Approach to Addressing Social Issues Using Administrative Data," IZA Policy Papers 199, Institute of Labor Economics (IZA).
    5. Estelle Dumas-Mallet & André Garenne & Thomas Boraud & François Gonon, 2020. "Does newspapers coverage influence the citations count of scientific publications? An analysis of biomedical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 413-427, April.
    6. Sesé Albert, 2018. "Discussing the Scientific Evidence Generation and Psychological Research Methods in Postmodern Societies," Journal for Perspectives of Economic Political and Social Integration, Sciendo, vol. 24(2), pages 9-30, December.
    7. Robert Kirkby, 2023. "Quantitative Macroeconomics: Lessons Learned from Fourteen Replications," Computational Economics, Springer;Society for Computational Economics, vol. 61(2), pages 875-896, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Estelle Dumas-Mallet & Katherine Button & Thomas Boraud & Marcus Munafo & François Gonon, 2016. "Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-20, June.
    2. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    3. Vivian Leung & Frédérik Rousseau-Blass & Guy Beauchamp & Daniel S J Pang, 2018. "ARRIVE has not ARRIVEd: Support for the ARRIVE (Animal Research: Reporting of in vivo Experiments) guidelines does not improve the reporting quality of papers in animal welfare, analgesia or anesthesi," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-13, May.
    4. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    5. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    6. Muradchanian, Jasmine & Hoekstra, Rink & Kiers, Henk & van Ravenzwaaij, Don, 2020. "How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics," MetaArXiv wvdjf, Center for Open Science.
    7. Louis Anthony (Tony) Cox, 2015. "Overcoming Learning Aversion in Evaluating and Managing Uncertain Risks," Risk Analysis, John Wiley & Sons, vol. 35(10), pages 1892-1910, October.
    8. Jeff Miller & Rolf Ulrich, 2019. "The quest for an optimal alpha," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-13, January.
    9. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    10. Stavroula Kousta & Christine Ferguson & Emma Ganley, 2016. "Meta-Research: Broadening the Scope of PLOS Biology," PLOS Biology, Public Library of Science, vol. 14(1), pages 1-2, January.
    11. Adriano Koshiyama & Nick Firoozye, 2019. "Avoiding Backtesting Overfitting by Covariance-Penalties: an empirical investigation of the ordinary and total least squares cases," Papers 1905.05023, arXiv.org.
    12. Louis Anthony (Tony) Cox, 2013. "Improving Causal Inferences in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 33(10), pages 1762-1771, October.
    13. Hannah Fraser & Tim Parker & Shinichi Nakagawa & Ashley Barnett & Fiona Fidler, 2018. "Questionable research practices in ecology and evolution," PLOS ONE, Public Library of Science, vol. 13(7), pages 1-16, July.
    14. Salandra, Rossella, 2018. "Knowledge dissemination in clinical trials: Exploring influences of institutional support and type of innovation on selective reporting," Research Policy, Elsevier, vol. 47(7), pages 1215-1228.
    15. David Chavalarias, 2017. "What’s wrong with Science?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 481-503, January.
    16. Andrew D Higginson & Marcus R Munafò, 2016. "Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions," PLOS Biology, Public Library of Science, vol. 14(11), pages 1-14, November.
    17. Markus Lehmkuhl & Nikolai Promies, 2020. "Frequency distribution of journalistic attention for scientific studies and scientific sources: An input–output analysis," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-20, November.
    18. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    19. Dean A Fergusson & Marc T Avey & Carly C Barron & Mathew Bocock & Kristen E Biefer & Sylvain Boet & Stephane L Bourque & Isidora Conic & Kai Chen & Yuan Yi Dong & Grace M Fox & Ronald B George & Neil , 2019. "Reporting preclinical anesthesia study (REPEAT): Evaluating the quality of reporting in the preclinical anesthesiology literature," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-15, May.
    20. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0172650. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.