IDEAS home Printed from https://ideas.repec.org/a/nat/nathum/v4y2020i4d10.1038_s41562-019-0787-z.html
   My bibliography  Save this article

Comparing meta-analyses and preregistered multiple-laboratory replication projects

Author

Listed:
  • Amanda Kvarven

    (University of Bergen)

  • Eirik Strømland

    (University of Bergen)

  • Magnus Johannesson

    (Stockholm School of Economics)

Abstract

Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications. We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results.

Suggested Citation

  • Amanda Kvarven & Eirik Strømland & Magnus Johannesson, 2020. "Comparing meta-analyses and preregistered multiple-laboratory replication projects," Nature Human Behaviour, Nature, vol. 4(4), pages 423-434, April.
    • Handle: RePEc:nat:nathum:v:4:y:2020:i:4:d:10.1038_s41562-019-0787-z
    DOI: 10.1038/s41562-019-0787-z
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41562-019-0787-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1038/s41562-019-0787-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Irsova, Zuzana & Bom, Pedro R. D. & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," EconStor Preprints 268683, ZBW - Leibniz Information Centre for Economics.
    2. Zuzana Irsova & Hristos Doucouliagos & Tomas Havranek & T. D. Stanley, 2024. "Meta‐analysis of social science research: A practitioner's guide," Journal of Economic Surveys, Wiley Blackwell, vol. 38(5), pages 1547-1566, December.
    3. Chris Doucouliagos & Jakob de Haan & Jan-Egbert Sturm, 2022. "What drives financial development? A Meta-regression analysis [A new database of financial reforms]," Oxford Economic Papers, Oxford University Press, vol. 74(3), pages 840-868.
    4. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    5. Fišar, Miloš & Cingl, Lubomír & Reggiani, Tommaso & Kundtová Klocová, Eva & Kundt, Radek & Krátký, Jan & Kostolanská, Katarína & Bencúrová, Petra & Pešková, Marie Kudličková & Marečková, Klára, 2023. "Ovulatory shift, hormonal changes, and no effects on incentivized decision-making," Journal of Economic Psychology, Elsevier, vol. 98(C).
    6. David Bilén & Anna Dreber & Magnus Johannesson, 2021. "Are women more generous than men? A meta-analysis," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 1-18, September.
    7. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    8. Abel Brodeur & Nikolai M. Cook & Jonathan S. Hartley & Anthony Heyes, 2024. "Do Preregistration and Preanalysis Plans Reduce p-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 2(3), pages 527-561.
    9. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawRxiv 2b5k4, Center for Open Science.
    10. Fossen, Frank M. & Neyse, Levent & Johannesson, Magnus & Dreber, Anna, 2022. "2D:4D and Self-Employment: A Preregistered Replication Study in a Large General Population Sample," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 46(1), pages 21-43.
    11. Kim, Bitna, 2022. "Publication bias: A “bird's-eye view” of meta-analytic practice in criminology and criminal justice," Journal of Criminal Justice, Elsevier, vol. 78(C).
    12. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawRxiv 2b5k4, Center for Open Science.
    13. Williams, Ralph I. & Clark, Leigh Anne & Clark, W. Randy & Raffo, Deana M., 2021. "Re-examining systematic literature review in management research: Additional benefits and execution protocols," European Management Journal, Elsevier, vol. 39(4), pages 521-533.
    14. Bartoš, František & Godmann, Henrik R., 2025. "A Comment on "A Systematic Review and Meta-Analysis of Transdiagnostic Cognitive Behavioural Therapies for Emotional Disorders"," I4R Discussion Paper Series 213, The Institute for Replication (I4R).
    15. Pavlo Blavatskyy & Valentyn Panchenko & Andreas Ortmann, 2023. "How common is the common-ratio effect?," Experimental Economics, Springer;Economic Science Association, vol. 26(2), pages 253-272, April.
    16. Doucouliagos, Hristos & Hinz, Thomas & Zigova, Katarina, 2022. "Bias and careers: Evidence from the aid effectiveness literature," European Journal of Political Economy, Elsevier, vol. 71(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:nathum:v:4:y:2020:i:4:d:10.1038_s41562-019-0787-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.