IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/v259t_v1.html

Preregistrations without Code do not Prevent P-Hacking

Author

Listed:
  • Röseler, Lukas

    (University of Münster)

Abstract

Preregistrations have been suggested as a tool to prevent p-hacking. Most researchers have used semi-structured preregistration templates to deprive themselves of researchers’ degrees of freedom. I show how such “narrative preregistrations” do not fully prevent p-hacking for even very simple statistical techniques (e.g., correlations) unless the analysis code is part of the preregistration.

Suggested Citation

  • Röseler, Lukas, 2025. "Preregistrations without Code do not Prevent P-Hacking," MetaArXiv v259t_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:v259t_v1
    DOI: 10.31219/osf.io/v259t_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/69219020b0dd4ce2fab5511f/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/v259t_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:osf:osfxxx:umq8d_v1 is not listed on IDEAS
    2. Abel Brodeur & Nikolai M. Cook & Jonathan S. Hartley & Anthony Heyes, 2024. "Do Preregistration and Preanalysis Plans Reduce p-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 2(3), pages 527-561.
    3. Wicherts, Jelte M. & Veldkamp, Coosje Lisabet Sterre & Augusteijn, Hilde & Bakker, Marjan & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2016. "Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking," OSF Preprints umq8d, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Reca Sarfati & Vod Vilfort, 2025. ""Post" Pre-Analysis Plans: Valid Inference for Non-Preregistered Specifications," Papers 2510.02507, arXiv.org.
    2. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    3. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    4. Maximilian M Mandl & Sabine Hoffmann & Sebastian Bieringer & Anna E Jacob & Marie Kraft & Simon Lemster & Anne-Laure Boulesteix, 2024. "Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices," PLOS Computational Biology, Public Library of Science, vol. 20(3), pages 1-10, March.
    5. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    6. repec:osf:metaar:s4b65_v1 is not listed on IDEAS
    7. repec:osf:metaar:jt5zf_v1 is not listed on IDEAS
    8. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).
    9. repec:osf:osfxxx:gv25c_v1 is not listed on IDEAS
    10. repec:osf:osfxxx:7dc6a_v1 is not listed on IDEAS
    11. Felix Holzmeister & Magnus Johannesson & Colin F. Camerer & Yiling Chen & Teck-Hua Ho & Suzanne Hoogeveen & Juergen Huber & Noriko Imai & Taisuke Imai & Lawrence Jin & Michael Kirchler & Alexander Ly , 2025. "Examining the replicability of online experiments selected by a decision market," Nature Human Behaviour, Nature, vol. 9(2), pages 316-330, February.
    12. repec:osf:metaar:2bj85_v1 is not listed on IDEAS
    13. Alexander Mielke, 2023. "Impact of dominance rank specification in dyadic interaction models," PLOS ONE, Public Library of Science, vol. 18(7), pages 1-19, July.
    14. Freuli, Francesca & Held, Leonhard & Heyard, Rachel, 2022. "Replication Success under Questionable Research Practices - A Simulation Study," I4R Discussion Paper Series 2, The Institute for Replication (I4R).
    15. Daniel McNeish, 2024. "Practical Implications of Sum Scores Being Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1148-1169, December.
    16. Thibaut Arpinon & Romain Espinosa, 2023. "A Practical Guide to Registered Reports for Economists," Post-Print halshs-03897719, HAL.
    17. Asanov, Anastasiya-Mariya & Asanov, Igor & Buenstorf, Guido & Kadriu, Valon & Schoch, Pia, 2025. "The Origins of Reporting Bias: Selective but Unbiased Reporting by Early-Career Researchers?," I4R Discussion Paper Series 225, The Institute for Replication (I4R).
    18. Lisa Spitzer & Stefanie Mueller, 2023. "Registered report: Survey on attitudes and experiences regarding preregistration in psychological research," PLOS ONE, Public Library of Science, vol. 18(3), pages 1-34, March.
    19. Shaw, Steven D. & Nave, Gideon, 2023. "Don't hate the player, hate the game: Realigning incentive structures to promote robust science and better scientific practices in marketing," Journal of Business Research, Elsevier, vol. 167(C).
    20. Gabriel Nova & Sander van Cranenburgh & Stephane Hess, 2024. "Understanding the decision-making process of choice modellers," Papers 2411.01704, arXiv.org, revised Jun 2025.
    21. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    22. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    23. Droz, Bénédicte & Buechel, Berno & Capra, Mónica & Chen, Xi & Nassar, Anis & Park, Seong Gyu & Xu, Jin & Zhang, Shanshan & Tasoff, Joshua, 2025. "Appetite for Ignorance: Does eating meat cause information avoidance about its harms?," European Economic Review, Elsevier, vol. 175(C).
    24. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArchive 2b5k4_v1, Center for Open Science.
    25. Denise Rousseau & Byeong Jo Kim & Ryan Splenda & Sarah Young & Jangbum Lee & Donna Beck, 2023. "Does chief executive compensation predict financial performance or inaccurate financial reporting in listed companies: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 19(4), December.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:v259t_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.