IDEAS home Printed from https://ideas.repec.org/a/plo/pbio00/1001609.html
   My bibliography  Save this article

Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases

Author

Listed:
  • Konstantinos K Tsilidis
  • Orestis A Panagiotou
  • Emily S Sena
  • Eleni Aretouli
  • Evangelos Evangelou
  • David W Howells
  • Rustam Al-Shahi Salman
  • Malcolm R Macleod
  • John P A Ioannidis

Abstract

: The evaluation of 160 meta-analyses of animal studies on potential treatments for neurological disorders reveals that the number of statistically significant results was too large to be true, suggesting biases. Animal studies generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials. We assessed whether there is evidence for excess statistical significance in results of animal studies on neurological disorders, suggesting biases. We used data from meta-analyses of interventions deposited in Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES). The number of observed studies with statistically significant results (O) was compared with the expected number (E), based on the statistical power of each study under different assumptions for the plausible effect size. We assessed 4,445 datasets synthesized in 160 meta-analyses on Alzheimer disease (n = 2), experimental autoimmune encephalomyelitis (n = 34), focal ischemia (n = 16), intracerebral hemorrhage (n = 61), Parkinson disease (n = 45), and spinal cord injury (n = 2). 112 meta-analyses (70%) found nominally (p≤0.05) statistically significant summary fixed effects. Assuming the effect size in the most precise study to be a plausible effect, 919 out of 4,445 nominally significant results were expected versus 1,719 observed (p

Suggested Citation

  • Konstantinos K Tsilidis & Orestis A Panagiotou & Emily S Sena & Eleni Aretouli & Evangelos Evangelou & David W Howells & Rustam Al-Shahi Salman & Malcolm R Macleod & John P A Ioannidis, 2013. "Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases," PLOS Biology, Public Library of Science, vol. 11(7), pages 1-10, July.
    • Handle: RePEc:plo:pbio00:1001609
    DOI: 10.1371/journal.pbio.1001609
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001609
    Download Restriction: no
    File URL: https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.1001609&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pbio.1001609?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. H Bart van der Worp & David W Howells & Emily S Sena & Michelle J Porritt & Sarah Rewell & Victoria O'Collins & Malcolm R Macleod, 2010. "Can Animal Models of Disease Reliably Inform Human Studies?," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-8, March.
    2. Emily S Sena & H Bart van der Worp & Philip M W Bath & David W Howells & Malcolm R Macleod, 2010. "Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy," PLOS Biology, Public Library of Science, vol. 8(3), pages 1-8, March.
    3. Carol Kilkenny & William J Browne & Innes C Cuthill & Michael Emerson & Douglas G Altman, 2010. "Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research," PLOS Biology, Public Library of Science, vol. 8(6), pages 1-5, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Oliver Braganza, 2020. "A simple model suggesting economically rational sample-size choice drives irreproducibility," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-19, March.
    2. Susanne Wieschowski & Diego S Silva & Daniel Strech, 2016. "Animal Study Registries: Results from a Stakeholder Analysis on Potential Strengths, Weaknesses, Facilitators, and Barriers," PLOS Biology, Public Library of Science, vol. 14(11), pages 1-12, November.
    3. Simone J Jonker & Theo P Menting & Michiel C Warlé & Merel Ritskes-Hoitinga & Kimberley E Wever, 2016. "Preclinical Evidence for the Efficacy of Ischemic Postconditioning against Renal Ischemia-Reperfusion Injury, a Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(3), pages 1-19, March.
    4. Ta-Chuan Yeh & Chih-Sung Liang & Chia-Kuang Tsai & Marco Solmi & Beny Lafer & Ping-Tao Tseng & Chih-Wei Hsu & Pao-Yen Lin & Joseph Firth & Brendon Stubbs & Lamiece Hassan & Michele Fornaro & Eduard Vi, 2022. "Neurological, Psychiatric, and Psychological Implications of the COVID-19 Pandemic: Protocol for a Large-Scale Umbrella Review of Observational Studies," IJERPH, MDPI, vol. 19(3), pages 1-9, February.
    5. Stephan B Bruns & John P A Ioannidis, 2016. "p-Curve and p-Hacking in Observational Research," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-13, February.
    6. Constance Holman & Sophie K Piper & Ulrike Grittner & Andreas Antonios Diamantaras & Jonathan Kimmelman & Bob Siegerink & Ulrich Dirnagl, 2016. "Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke," PLOS Biology, Public Library of Science, vol. 14(1), pages 1-12, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Constance Holman & Sophie K Piper & Ulrike Grittner & Andreas Antonios Diamantaras & Jonathan Kimmelman & Bob Siegerink & Ulrich Dirnagl, 2016. "Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke," PLOS Biology, Public Library of Science, vol. 14(1), pages 1-12, January.
    2. Kimberley E Wever & Carlijn R Hooijmans & Niels P Riksen & Thomas B Sterenborg & Emily S Sena & Merel Ritskes-Hoitinga & Michiel C Warlé, 2015. "Determinants of the Efficacy of Cardiac Ischemic Preconditioning: A Systematic Review and Meta-Analysis of Animal Studies," PLOS ONE, Public Library of Science, vol. 10(11), pages 1-17, November.
    3. Nathalie Percie du Sert & Viki Hurst & Amrita Ahluwalia & Sabina Alam & Marc T Avey & Monya Baker & William J Browne & Alejandra Clark & Innes C Cuthill & Ulrich Dirnagl & Michael Emerson & Paul Garne, 2020. "The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research," PLOS Biology, Public Library of Science, vol. 18(7), pages 1-12, July.
    4. David Baker & Katie Lidster & Ana Sottomayor & Sandra Amor, 2014. "Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies," PLOS Biology, Public Library of Science, vol. 12(1), pages 1-6, January.
    5. Susanne Wieschowski & Diego S Silva & Daniel Strech, 2016. "Animal Study Registries: Results from a Stakeholder Analysis on Potential Strengths, Weaknesses, Facilitators, and Barriers," PLOS Biology, Public Library of Science, vol. 14(11), pages 1-12, November.
    6. Jennifer A Hirst & Jeremy Howick & Jeffrey K Aronson & Nia Roberts & Rafael Perera & Constantinos Koshiaris & Carl Heneghan, 2014. "The Need for Randomization in Animal Trials: An Overview of Systematic Reviews," PLOS ONE, Public Library of Science, vol. 9(6), pages 1-11, June.
    7. Carol Kilkenny & William J Browne & Innes C Cuthill & Michael Emerson & Douglas G Altman, 2010. "Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research," PLOS Biology, Public Library of Science, vol. 8(6), pages 1-5, June.
    8. Gillian L Currie & Helena N Angel-Scott & Lesley Colvin & Fala Cramond & Kaitlyn Hair & Laila Khandoker & Jing Liao & Malcolm Macleod & Sarah K McCann & Rosie Morland & Nicki Sherratt & Robert Stewart, 2019. "Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-34, May.
    9. Ana Antonic & Emily S Sena & Jennifer S Lees & Taryn E Wills & Peta Skeers & Peter E Batchelor & Malcolm R Macleod & David W Howells, 2013. "Stem Cell Transplantation in Traumatic Spinal Cord Injury: A Systematic Review and Meta-Analysis of Animal Studies," PLOS Biology, Public Library of Science, vol. 11(12), pages 1-14, December.
    10. Julie E. Goodman & Catherine Petito Boyce & Sonja N. Sax & Leslie A. Beyer & Robyn L. Prueitt, 2015. "Rethinking Meta‐Analysis: Applications for Air Pollution Data and Beyond," Risk Analysis, John Wiley & Sons, vol. 35(6), pages 1017-1039, June.
    11. Airi Jo-Watanabe & Toshiki Inaba & Takahiro Osada & Ryota Hashimoto & Tomohiro Nishizawa & Toshiaki Okuno & Sayoko Ihara & Kazushige Touhara & Nobutaka Hattori & Masatsugu Oh-Hora & Osamu Nureki & Tak, 2024. "Bicarbonate signalling via G protein-coupled receptor regulates ischaemia-reperfusion injury," Nature Communications, Nature, vol. 15(1), pages 1-18, December.
    12. Dean A Fergusson & Marc T Avey & Carly C Barron & Mathew Bocock & Kristen E Biefer & Sylvain Boet & Stephane L Bourque & Isidora Conic & Kai Chen & Yuan Yi Dong & Grace M Fox & Ronald B George & Neil , 2019. "Reporting preclinical anesthesia study (REPEAT): Evaluating the quality of reporting in the preclinical anesthesiology literature," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-15, May.
    13. Lauralyn A McIntyre & David Moher & Dean A Fergusson & Katrina J Sullivan & Shirley H J Mei & Manoj Lalu & John Marshall & Malcolm Mcleod & Gilly Griffin & Jeremy Grimshaw & Alexis Turgeon & Marc T Av, 2016. "Efficacy of Mesenchymal Stromal Cell Therapy for Acute Lung Injury in Preclinical Animal Models: A Systematic Review," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-16, January.
    14. M Polyakova & M L Schroeter & B M Elzinga & S Holiga & P Schoenknecht & E R de Kloet & M L Molendijk, 2015. "Brain-Derived Neurotrophic Factor and Antidepressive Effect of Electroconvulsive Therapy: Systematic Review and Meta-Analyses of the Preclinical and Clinical Literature," PLOS ONE, Public Library of Science, vol. 10(11), pages 1-18, November.
    15. Robyn M. Lucas & Rachael M. Rodney Harris, 2018. "On the Nature of Evidence and ‘Proving’ Causality: Smoking and Lung Cancer vs. Sun Exposure, Vitamin D and Multiple Sclerosis," IJERPH, MDPI, vol. 15(8), pages 1-13, August.
    16. Bettina Bert & Céline Heinl & Justyna Chmielewska & Franziska Schwarz & Barbara Grune & Andreas Hensel & Matthias Greiner & Gilbert Schönfelder, 2019. "Refining animal research: The Animal Study Registry," PLOS Biology, Public Library of Science, vol. 17(10), pages 1-12, October.
    17. Xiao-meng Xu & Guang-yan Cai & Ru Bu & Wen-juan Wang & Xue-yuan Bai & Xue-feng Sun & Xiang-mei Chen, 2015. "Beneficial Effects of Caloric Restriction on Chronic Kidney Disease in Rodent Models: A Meta-Analysis and Systematic Review," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-15, December.
    18. Zhongwei Xu & Bingze Xu & Susanna L. Lundström & Àlex Moreno-Giró & Danxia Zhao & Myriam Martin & Erik Lönnblom & Qixing Li & Alexander Krämer & Changrong Ge & Lei Cheng & Bibo Liang & Dongmei Tong & , 2023. "A subset of type-II collagen-binding antibodies prevents experimental arthritis by inhibiting FCGR3 signaling in neutrophils," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    19. Bossert, Leonie & Hagendorff, Thilo, 2021. "Animals and AI. The role of animals in AI research and application – An overview and ethical evaluation," Technology in Society, Elsevier, vol. 67(C).
    20. Vivian Leung & Frédérik Rousseau-Blass & Guy Beauchamp & Daniel S J Pang, 2018. "ARRIVE has not ARRIVEd: Support for the ARRIVE (Animal Research: Reporting of in vivo Experiments) guidelines does not improve the reporting quality of papers in animal welfare, analgesia or anesthesi," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-13, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pbio00:1001609. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosbiology (email available below). General contact details of provider: https://journals.plos.org/plosbiology/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.