IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0233107.html
   My bibliography  Save this article

Reproducibility of individual effect sizes in meta-analyses in psychology

Author

Listed:
  • Esther Maassen
  • Marcel A L M van Assen
  • Michèle B Nuijten
  • Anton Olsson-Collentine
  • Jelte M Wicherts

Abstract

To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis. Results showed that almost half (k = 224) of all sampled primary effect sizes could not be reproduced based on the reported information in the meta-analysis, mostly because of incomplete or missing information on how effect sizes from primary studies were selected and computed. Overall, this led to small discrepancies in the computation of mean effect sizes, confidence intervals and heterogeneity estimates in 13 out of 33 meta-analyses. We provide recommendations to improve transparency in the reporting of the entire meta-analytic process, including the use of preregistration, data and workflow sharing, and explicit coding practices.

Suggested Citation

  • Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    • Handle: RePEc:plo:pone00:0233107
    DOI: 10.1371/journal.pone.0233107
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0233107
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0233107&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0233107?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    2. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    3. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    4. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    5. Matthew J Page & Larissa Shamseer & Douglas G Altman & Jennifer Tetzlaff & Margaret Sampson & Andrea C Tricco & Ferrán Catalá-López & Lun Li & Emma K Reid & Rafael Sarkis-Onofre & David Moher, 2016. "Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study," PLOS Medicine, Public Library of Science, vol. 13(5), pages 1-30, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    2. Heidi Seibold & Severin Czerny & Siona Decke & Roman Dieterle & Thomas Eder & Steffen Fohr & Nico Hahn & Rabea Hartmann & Christoph Heindl & Philipp Kopper & Dario Lepke & Verena Loidl & Maximilian Ma, 2021. "A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-15, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    2. Neal R. Haddaway & Matthew J. Page & Chris C. Pritchard & Luke A. McGuinness, 2022. "PRISMA2020: An R package and Shiny app for producing PRISMA 2020‐compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(2), June.
    3. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.
    4. Mahesh Shumsher Rughooputh & Rui Zeng & Ying Yao, 2015. "Protein Diet Restriction Slows Chronic Kidney Disease Progression in Non-Diabetic and in Type 1 Diabetic Patients, but Not in Type 2 Diabetic Patients: A Meta-Analysis of Randomized Controlled Trials ," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-17, December.
    5. Christopher Winchester & Kelsey E. Medeiros, 2023. "In Bounds but Out of the Box: A Meta-Analysis Clarifying the Effect of Ethicality on Creativity," Journal of Business Ethics, Springer, vol. 183(3), pages 713-743, March.
    6. Kelly R Moran & Sara Y Del Valle, 2016. "A Meta-Analysis of the Association between Gender and Protective Behaviors in Response to Respiratory Epidemics and Pandemics," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-25, October.
    7. Eloy López-Meneses & Esteban Vázquez-Cano & Mariana-Daniela González-Zamar & Emilio Abad-Segura, 2020. "Socioeconomic Effects in Cyberbullying: Global Research Trends in the Educational Context," IJERPH, MDPI, vol. 17(12), pages 1-31, June.
    8. Sandra Feijóo & Raquel Rodríguez-Fernández, 2021. "A Meta-Analytical Review of Gender-Based School Bullying in Spain," IJERPH, MDPI, vol. 18(23), pages 1-13, December.
    9. Xizheng Xu & Zhiqiang Liu & Shaoying Gong & Yunpeng Wu, 2022. "The Relationship between Empathy and Attachment in Children and Adolescents: Three-Level Meta-Analyses," IJERPH, MDPI, vol. 19(3), pages 1-18, January.
    10. Kathrin Wunsch & Janis Fiedler & Philip Bachert & Alexander Woll, 2021. "The Tridirectional Relationship among Physical Activity, Stress, and Academic Performance in University Students: A Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 18(2), pages 1-18, January.
    11. Alan da Silveira Fleck & Margaux L. Sadoine & Stéphane Buteau & Eva Suarthana & Maximilien Debia & Audrey Smargiassi, 2021. "Environmental and Occupational Short-Term Exposure to Airborne Particles and FEV 1 and FVC in Healthy Adults: A Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 18(20), pages 1-19, October.
    12. Evangelos Danopoulos & Maureen Twiddy & Jeanette M Rotchell, 2020. "Microplastic contamination of drinking water: A systematic review," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-23, July.
    13. Claudia Menne-Lothmann & Wolfgang Viechtbauer & Petra Höhn & Zuzana Kasanova & Simone P Haller & Marjan Drukker & Jim van Os & Marieke Wichers & Jennifer Y F Lau, 2014. "How to Boost Positive Interpretations? A Meta-Analysis of the Effectiveness of Cognitive Bias Modification for Interpretation," PLOS ONE, Public Library of Science, vol. 9(6), pages 1-26, June.
    14. Wei-Cheng Chang & Chin Lin & Cho-Hao Lee & Tzu-Ling Sung & Tao-Hsin Tung & Jorn-Hon Liu, 2017. "Vitrectomy with or without internal limiting membrane peeling for idiopathic epiretinal membrane: A meta-analysis," PLOS ONE, Public Library of Science, vol. 12(6), pages 1-18, June.
    15. Christopher Hansen & Holger Steinmetz & Jörn Block, 2022. "How to conduct a meta-analysis in eight steps: a practical guide," Management Review Quarterly, Springer, vol. 72(1), pages 1-19, February.
    16. Shaylea Badovinac & Jodi Martin & Camille Guérin-Marion & Monica O’Neill & Rebecca Pillai Riddell & Jean-François Bureau & Rebecca Spiegel, 2018. "Associations between mother-preschooler attachment and maternal depression symptoms: A systematic review and meta-analysis," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-27, October.
    17. Fazel, Seena & Burghart, Matthias & Fanshawe, Thomas & Gil, Sharon Danielle & Monahan, John & Yu, Rongqin, 2022. "The predictive performance of criminal risk assessment tools used at sentencing: Systematic review of validation studies," Journal of Criminal Justice, Elsevier, vol. 81(C).
    18. Amro Qaddoura & Payam Yazdan-Ashoori & Conrad Kabali & Lehana Thabane & R Brian Haynes & Stuart J Connolly & Harriette Gillian Christine Van Spall, 2015. "Efficacy of Hospital at Home in Patients with Heart Failure: A Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    19. Joseph Pozsgai-Alvarez & Iván Pastor Sanz, 2021. "Mapping the (anti-)corruption field: key topics and changing trends, 1968–2020," Journal of Computational Social Science, Springer, vol. 4(2), pages 851-881, November.
    20. Boshra H. Namin & Torvald Øgaard & Jo Røislien, 2021. "Workplace Incivility and Turnover Intention in Organizations: A Meta-Analytic Review," IJERPH, MDPI, vol. 19(1), pages 1-19, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0233107. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.