IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/cd5j9.html
   My bibliography  Save this paper

Observing Many Researchers using the Same Data and Hypothesis Reveals a Hidden Universe of Data Analysis

Author

Listed:
  • Breznau, Nate

    (University of Bremen)

  • Rinke, Eike Mark

    (University of Leeds)

  • Wuttke, Alexander

    (University of Mannheim)

  • Adem, Muna
  • Adriaans, Jule
  • Alvarez-Benjumea, Amalia

    (Max Planck Institute for Research on collective goods)

  • Andersen, Henrik Kenneth
  • Auer, Daniel
  • Azevedo, Flavio

    (Cologne University)

  • Bahnsen, Oke

Abstract

Findings from 162 researchers in 73 teams testing the same hypothesis with the same data reveal a universe of unique analytical possibilities leading to a broad range of results and conclusions. Surprisingly, the outcome variance mostly cannot be explained by variations in researchers’ modeling decisions or prior beliefs. Each of the 1,261 test models submitted by the teams was ultimately a unique combination of data-analytical steps. Because the noise generated in this crowdsourced research mostly cannot be explained using myriad meta-analytic methods, we conclude that idiosyncratic researcher variability is a threat to the reliability of scientific findings. This highlights the complexity and ambiguity inherent in the scientific data analysis process that needs to be taken into account in future efforts to assess and improve the credibility of scientific work.

Suggested Citation

  • Breznau, Nate & Rinke, Eike Mark & Wuttke, Alexander & Adem, Muna & Adriaans, Jule & Alvarez-Benjumea, Amalia & Andersen, Henrik Kenneth & Auer, Daniel & Azevedo, Flavio & Bahnsen, Oke, 2021. "Observing Many Researchers using the Same Data and Hypothesis Reveals a Hidden Universe of Data Analysis," MetaArXiv cd5j9, Center for Open Science.
    • Handle: RePEc:osf:metaar:cd5j9
    DOI: 10.31219/osf.io/cd5j9
    as

    Download full text from publisher

    File URL: https://osf.io/download/605b74e97d552b00ed1b750b/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/cd5j9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    2. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    3. Amy Orben & Andrew K. Przybylski, 2019. "The association between adolescent well-being and digital technology use," Nature Human Behaviour, Nature, vol. 3(2), pages 173-182, February.
    4. Rosseel, Yves, 2012. "lavaan: An R Package for Structural Equation Modeling," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i02).
    5. Raphael Silberzahn & Eric L. Uhlmann, 2015. "Crowdsourced research: Many hands make tight work," Nature, Nature, vol. 526(7572), pages 189-191, October.
    6. Brady, David & Finnigan, Ryan, 2014. "Does Immigration Undermine Public Support for Social Policy?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 79(1), pages 17-42.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Goldman, Daniel S, 2021. "A Reply to Breznau, Rinke, and Wuttke on the Discordant Outcomes of Hypothesis Testing," MetaArXiv qs5gm, Center for Open Science.
    2. Ruohuang Jiao & Wojtek Przepiorka & Vincent Buskens, 2022. "Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach," Journal of Computational Social Science, Springer, vol. 5(1), pages 1041-1067, May.
    3. Nick Huntington-Klein & Claus Pörtner & Yubraj Acharya & Matus Adamkovic & Joop Adema & Lameck Ondieki Agasa & Imtiaz Ahmad & Mevlude Akbulut-Yuksel & Martin Eckhoff Andresen & David Angenendt & José-, 2025. "The Sources of Researcher Variation in Economics," Working Papers hal-05187084, HAL.
    4. repec:osf:metaar:qs5gm_v1 is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alesina, Alberto & Murard, Elie & Rapoport, Hillel, 2019. "Immigration and Preferences for Redistribution in Europe," IZA Discussion Papers 12130, Institute of Labor Economics (IZA).
    2. Amy Orben & Andrew K. Przybylski & Sarah-Jayne Blakemore & Rogier A. Kievit, 2022. "Windows of developmental sensitivity to social media," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    3. Nate Breznau & Eike Mark Rinke & Alexander Wuttke & Hung H. V. Nguyen & Muna Adem & Jule Adriaans & Amalia Alvarez-Benjumea & Henrik K. Andersen & Daniel Auer & Flavio Azevedo & Oke Bahnsen & Dave Bal, 2022. "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 119(44), pages 2203150119-, November.
    4. Naoki Sudo, 2020. "Two Types of Support for Redistribution of Wealth: Consistent and Inconsistent Policy Preferences," Societies, MDPI, vol. 10(2), pages 1-18, June.
    5. Chantie Charissa Luijten & Daphne Bongardt & Anna Petra Nieboer, 2022. "The Roles of Social Media Use and Friendship Quality in Adolescents’ Internalizing Problems and Well-being," Journal of Happiness Studies, Springer, vol. 23(7), pages 3161-3178, October.
    6. Sonia Nawrocka & Hans De Witte & Margherita Pasini & Margherita Brondino, 2023. "A Person-Centered Approach to Job Insecurity: Is There a Reciprocal Relationship between the Quantitative and Qualitative Dimensions of Job Insecurity?," IJERPH, MDPI, vol. 20(7), pages 1-27, March.
    7. Christopher Hoy & Franziska Mager, 2019. "Why are relatively poor people not more supportive of redistribution? Evidence from a survey experiment across 10 countries," Working Papers 489, ECINEQ, Society for the Study of Economic Inequality.
    8. Diogo G. C. Britto & Alexandre Fonseca & Paolo Pinotti & Breno Sampaio & Lucas Warwar, 2022. "Intergenerational Mobility in the Land of Inequality," CESifo Working Paper Series 10004, CESifo.
    9. Lea Marie Heidbreder & Josephine Tröger & Manfred Schmitt, 2023. "Exploring the psychological antecedents of private and public sphere behaviours to reduce household plastic consumption," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 25(4), pages 3405-3428, April.
    10. Md. Mominur Rahman & Bilkis Akhter, 2021. "The impact of investment in human capital on bank performance: evidence from Bangladesh," Future Business Journal, Springer, vol. 7(1), pages 1-13, December.
    11. Olivier Coibion & Yuriy Gorodnichenko & Michael Weber, 2020. "Political Polarization and Expected Economic Outcomes," Working Papers 2020-158, Becker Friedman Institute for Research In Economics.
    12. Hampus Persson & Fredrik Björklund & Martin Bäckström, 2025. "How Social Desirability Influences the Relationship between Measures of Personality and Key Constructs in Positive Psychology," Journal of Happiness Studies, Springer, vol. 26(3), pages 1-17, March.
    13. Darren Haywood & Frank D. Baughman & Barbara A. Mullan & Karen R. Heslop, 2021. "One p -Factor for All? Exploring the Applicability of Structural Models of Psychopathology within Subgroups of a Population," IJERPH, MDPI, vol. 18(13), pages 1-22, July.
    14. Masashi Soga & Kevin J. Gaston & Yuichi Yamaura & Kiyo Kurisu & Keisuke Hanaki, 2016. "Both Direct and Vicarious Experiences of Nature Affect Children’s Willingness to Conserve Biodiversity," IJERPH, MDPI, vol. 13(6), pages 1-12, May.
    15. Knell, Markus & Stix, Helmut, 2020. "Perceptions of inequality," European Journal of Political Economy, Elsevier, vol. 65(C).
    16. Deutsch, Eliza S. & Alameddine, Ibrahim & Qian, Song S., 2020. "Using structural equation modeling to better understand microcystis biovolume dynamics in a mediterranean hypereutrophic reservoir," Ecological Modelling, Elsevier, vol. 435(C).
    17. César Merino-Soto & Gina Chávez-Ventura & Verónica López-Fernández & Guillermo M. Chans & Filiberto Toledano-Toledano, 2022. "Learning Self-Regulation Questionnaire (SRQ-L): Psychometric and Measurement Invariance Evidence in Peruvian Undergraduate Students," Sustainability, MDPI, vol. 14(18), pages 1-17, September.
    18. Hippolyte d’Albis & Ikpidi Badji, 2021. "Intergenerational equity by educational attainments in France," Journal of Population Research, Springer, vol. 38(4), pages 339-365, December.
    19. Solène Lefebvre du Grosriez & Sandrine Isoard-Gautheur & Mariya Yukhymenko-Lescroart & Philippe Sarrazin, 2024. "The adapted French version of the Academic and Athletic Identity Scale (AAIS-FR): Evidence of validity and reliability and relationships with sport well-being," PLOS ONE, Public Library of Science, vol. 19(5), pages 1-20, May.
    20. Corinne E Zachry & Le Vy Phan & Laura E R Blackie & Eranda Jayawickreme, 2018. "Situation-Based Contingencies Underlying Wisdom-Content Manifestations: Examining Intellectual Humility in Daily Life," The Journals of Gerontology: Series B, The Gerontological Society of America, vol. 73(8), pages 1404-1415.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:cd5j9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.