IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/315353.html
   My bibliography  Save this article

The reliability of replications: a study in computational reproductions

Author

Listed:
  • Breznau, Nate
  • Rinke, Eike Mark
  • Wuttke, Alexander
  • Adem, Muna
  • Adriaans, Jule
  • Akdeniz, Esra
  • Alvarez-Benjumea, Amalia
  • Andersen, Henrik K.
  • Auer, Daniel
  • Azevedo, Flavio
  • Bahnsen, Oke
  • Bai, Ling
  • Balzer, Dave
  • Bauer, Paul C.
  • Bauer, Gerrit
  • Baumann, Markus
  • Baute, Sharon
  • Benoit, Verena
  • Bernauer, Julian
  • Berning, Carl
  • Berthold, Anna
  • Bethke, Felix S.
  • Biegert, Thomas
  • Blinzler, Katharina
  • Blumenberg, Johannes N.
  • Bobzien, Licia
  • Bohman, Andrea
  • Bol, Thijs
  • Bostic, Amie
  • Brzozowska, Zuzanna
  • Burgdorf, Katharina
  • Burger, Kaspar
  • Busch, Kathrin
  • Castillo, Juan-Carlos
  • Chan, Nathan
  • Christmann, Pablo
  • Connelly, Roxanne
  • Czymara, Christian S.
  • Damian, Elena
  • de Rooij, Eline A.
  • Ecker, Alejandro
  • Edelmann, Achim
  • Eder, Christina
  • Eger, Maureen A.
  • Ellerbrock, Simon
  • Forke, Anna
  • Forster, Andrea
  • Freire, Danilo
  • Gaasendam, Chris
  • Gavras, Konstantin
  • Gayle, Vernon
  • Gessler, Theresa
  • Gnambs, Timo
  • Godefroidt, Amélie
  • Grömping, Max
  • Groß, Martin
  • Gruber, Stefan
  • Gummer, Tobias
  • Hadjar, Andreas
  • Halbherr, Verena
  • Heisig, Jan Paul
  • Hellmeier, Sebastian
  • Heyne, Stefanie
  • Hirsch, Magdalena
  • Hjerm, Mikael
  • Hochman, Oshrat
  • Höffler, Jan H.
  • Hövermann, Andreas
  • Hunger, Sophia
  • Hunkler, Christian
  • Huth-Stöckle, Nora
  • Ignácz, Zsófia S.
  • Israel, Sabine
  • Jacobs, Laura
  • Jacobsen, Jannes
  • Jaeger, Bastian
  • Jungkunz, Sebastian
  • Jungmann, Nils
  • Kanjana, Jennifer
  • Kauff, Mathias
  • Khan, Salman
  • Khatua, Sayak
  • Kleinert, Manuel
  • Klinger, Julia
  • Kolb, Jan-Philipp
  • Kołczyńska, Marta
  • Kuk, John
  • Kunißen, Katharina
  • Kurti Sinatra, Dafina
  • Langenkamp, Alexander
  • Lee, Robin C.
  • Lersch, Philipp M.
  • Liu, David
  • Löbel, Lea-Maria
  • Lutscher, Philipp
  • Mader, Matthias
  • Madia, Joan E.
  • Malancu, Natalia
  • Maldonado, Luis
  • Marahrens, Helge
  • Martin, Nicole
  • Martinez, Paul
  • Mayerl, Jochen
  • Mayorga, Oscar J.
  • McDonnell, Robert
  • McManus, Patricia
  • McWagner, Kyle
  • Meeusen, Cecil
  • Meierrieks, Daniel
  • Mellon, Jonathan
  • Merhout, Friedolin
  • Merk, Samuel
  • Meyer, Daniel
  • Micheli, Leticia
  • Mijs, Jonathan
  • Moya, Cristóbal
  • Neunhoeffer, Marcel
  • Nüst, Daniel
  • Nygård, Olav
  • Ochsenfeld, Fabian
  • Otte, Gunnar
  • Pechenkina, Anna
  • Pickup, Mark
  • Prosser, Christopher
  • Raes, Louis
  • Ralston, Kevin
  • Ramos, Miguel
  • Reichert, Frank
  • Roets, Arne
  • Rogers, Jonathan
  • Ropers, Guido
  • Samuel, Robin
  • Sand, Gregor
  • Sanhueza Petrarca, Constanza
  • Schachter, Ariela
  • Schaeffer, Merlin
  • Schieferdecker, David
  • Schlueter, Elmar
  • Schmidt, Katja
  • Schmidt, Regine
  • Schmidt-Catran, Alexander
  • Schmiedeberg, Claudia
  • Schneider, Jürgen
  • Schoonvelde, Martijn
  • Schulte-Cloos, Julia
  • Schumann, Sandy
  • Schunck, Reinhard
  • Seuring, Julian
  • Silber, Henning
  • Sleegers, Willem
  • Sonntag, Nico
  • Staudt, Alexander
  • Steiber, Nadia
  • Steiner, Nils D.
  • Sternberg, Sebastian
  • Stiers, Dieter
  • Stojmenovska, Dragana
  • Storz, Nora
  • Striessnig, Erich
  • Stroppe, Anne-Kathrin
  • Suchow, Jordan W.
  • Teltemann, Janna
  • Tibajev, Andrey
  • Tung, Brian
  • Vagni, Giacomo
  • Van Assche, Jasper
  • van der Linden, Meta
  • van der Noll, Jolanda
  • Van Hootegem, Arno
  • Vogtenhuber, Stefan
  • Voicu, Bogdan
  • Wagemans, Fieke
  • Wehl, Nadja
  • Werner, Hannah
  • Wiernik, Brenton M.
  • Winter, Fabian
  • Wolf, Christof
  • Wu, Cary
  • Yamada, Yuki
  • Zakula, Björn
  • Zhang, Nan
  • Ziller, Conrad
  • Zins, Stefan
  • Żółtak, Tomasz
  • Nguyen, Hung H.V.

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p -value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.

Suggested Citation

  • Breznau, Nate & Rinke, Eike Mark & Wuttke, Alexander & Adem, Muna & Adriaans, Jule & Akdeniz, Esra & Alvarez-Benjumea, Amalia & Andersen, Henrik K. & Auer, Daniel & Azevedo, Flavio & Bahnsen, Oke & Ba, 2025. "The reliability of replications: a study in computational reproductions," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 12(3), pages 1-23.
  • Handle: RePEc:zbw:espost:315353
    DOI: 10.1098/rsos.241038
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/315353/1/Full-text-article-Breznau-et-al-The-reliability-of-replications.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.1098/rsos.241038?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Thu-Mai Christian & Amanda Gooch & Todd Vision & Elizabeth Hull, 2020. "Journal data policies: Exploring how the understanding of editors and authors corresponds to the policies themselves," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-15, March.
    2. Brady, David & Finnigan, Ryan, 2014. "Does Immigration Undermine Public Support for Social Policy?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 79(1), pages 17-42.
    3. Nate Breznau & Eike Mark Rinke & Alexander Wuttke & Hung H. V. Nguyen & Muna Adem & Jule Adriaans & Amalia Alvarez-Benjumea & Henrik K. Andersen & Daniel Auer & Flavio Azevedo & Oke Bahnsen & Dave Bal, 2022. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty," Decision Analysis, INFORMS, vol. 119(44), pages 2203150119-, November.
    4. Breznau, Nate & Rinke, Eike Mark & Wuttke, Alexander & Nguyen, Hung H. V. & Adem, Muna & Adriaans, Jule & Alvarez-Benjumea, Amalia & Andersen, Henrik K. & Auer, Daniel & Azevedo, Flavio & Bahnsen, Oke, 2022. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 119(44), pages 1-8.
    5. Sam Parsons & Flávio Azevedo & Mahmoud M. Elsherif & Samuel Guay & Owen N. Shahim & Gisela H. Govaart & Emma Norris & Aoife O’Mahony & Adam J. Parker & Ana Todorovic & Charlotte R. Pennington & Elias , 2022. "A community-sourced glossary of open scholarship terms," Nature Human Behaviour, Nature, vol. 6(3), pages 312-318, March.
    6. Chung-hong Chan & David Schoch, 2023. "rang: Reconstructing reproducible R computational environments," PLOS ONE, Public Library of Science, vol. 18(6), pages 1-14, June.
    7. Justin T. Pickett, 2020. "The Stewart Retractions: A Quantitative and Qualitative Analysis," Econ Journal Watch, Econ Journal Watch, vol. 17(1), pages 152–190-1, March.
    8. Monya Baker, 2016. "1,500 scientists lift the lid on reproducibility," Nature, Nature, vol. 533(7604), pages 452-454, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Deer, Lachlan & Adler, Susanne J. & Datta, Hannes & Mizik, Natalie & Sarstedt, Marko, 2025. "Toward open science in marketing research," International Journal of Research in Marketing, Elsevier, vol. 42(1), pages 212-233.
    2. Cantone, Giulio Giacomo & Tomaselli, Venera, 2023. "A Multiversal Model of Vibration of Effects of the Equitable and Sustainable Well-Being (BES) on Fertility," MetaArXiv z5msx, Center for Open Science.
    3. Christoph Huber & Anna Dreber & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Utz Weitzel & Miguel Abellán & Xeniya Adayeva & Fehime Ceren Ay & Kai Barron & Zachariah Berry & Werner Bönte , 2023. "Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(23), pages 2215572120-, June.
    4. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    5. Alessio Lachi & Josep Llach & Jordi Perramon & Michela Baccini & Andrea Saltelli, 2025. "Robustification of structural equation modelling via global sensitivity analysis," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 34(2), pages 211-236, May.
    6. Giulio Giacomo Cantone & Venera Tomaselli, 2024. "Theory and methods of the multiverse: an application for panel-based models," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(2), pages 1447-1480, April.
    7. Yu, Bin, 2023. "What is uncertainty in today’s practice of data science?," Journal of Econometrics, Elsevier, vol. 237(1).
    8. Cantone, Giulio Giacomo & Tomaselli, Venera, 2023. "Multiversal Methods and Applications," MetaArXiv ukvw7, Center for Open Science.
    9. repec:osf:metaar:z5msx_v1 is not listed on IDEAS
    10. Kleber Neves & Pedro B Tan & Olavo B Amaral, 2022. "Are most published research findings false in a continuous universe?," PLOS ONE, Public Library of Science, vol. 17(12), pages 1-18, December.
    11. Benjamin Elsner & Jeff Concannon, 2020. "Immigration and Redistribution," Working Papers 202024, School of Economics, University College Dublin.
    12. Alesina, Alberto & Murard, Elie & Rapoport, Hillel, 2019. "Immigration and Preferences for Redistribution in Europe," IZA Discussion Papers 12130, Institute of Labor Economics (IZA).
    13. Dennis Bontempi & Leonard Nuernberg & Suraj Pai & Deepa Krishnaswamy & Vamsi Thiriveedhi & Ahmed Hosny & Raymond H. Mak & Keyvan Farahani & Ron Kikinis & Andrey Fedorov & Hugo J. W. L. Aerts, 2024. "End-to-end reproducible AI pipelines in radiology using the cloud," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    14. Tom VanHeuvelen & Kathy Copas, 2018. "The Intercohort Dynamics of Support for Redistribution in 54 Countries, 1985–2017," Societies, MDPI, vol. 8(3), pages 1-22, August.
    15. Chai, Daniel & Ali, Searat & Brosnan, Mark & Hasso, Tim, 2024. "Understanding researchers' perceptions and experiences in finance research replication studies: A pre-registered report," Pacific-Basin Finance Journal, Elsevier, vol. 86(C).
    16. Céline Piton, 2022. "The labour market performance of vulnerable groups: towards a better understanding of the main driving forces," ULB Institutional Repository 2013/352519, ULB -- Universite Libre de Bruxelles.
    17. JaeYoul Shin, 2018. "Relative Deprivation, Satisfying Rationality, and Support for Redistribution," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 140(1), pages 35-56, November.
    18. Roberto Savona & Cristina Maria Alberini & Lucia Alessi & Iacopo Baussano & Petros Dellaportas & Ranieri Guerra & Sean Khozin & Andrea Modena & Sergio Pecorelli & Guido Rasi & Paolo Daniele Siviero & , 2023. "Towards a Framework for a New Research Ecosystem," Papers 2312.07065, arXiv.org, revised Dec 2023.
    19. Pascal Achard & Sigrid Suetens, 2023. "The causal effect of ethnic diversity on support for redistribution and the role of discrimination," Journal of Economic Surveys, Wiley Blackwell, vol. 37(5), pages 1678-1696, December.
    20. repec:plo:pcbi00:1005412 is not listed on IDEAS
    21. repec:osf:metaar:ukvw7_v1 is not listed on IDEAS
    22. Peter Harremoës, 2019. "Replication Papers," Publications, MDPI, vol. 7(3), pages 1-8, July.
    23. Joonghyun Kwak & Michael Wallace, 2018. "The Impact of the Great Recession on Perceived Immigrant Threat: A Cross-National Study of 22 Countries," Societies, MDPI, vol. 8(3), pages 1-23, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:315353. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.