IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/s4b65.html
   My bibliography  Save this paper

Replication success under questionable research practices – a simulation study

Author

Listed:
  • Freuli, Francesca
  • Held, Leonhard
  • Heyard, Rachel

Abstract

Increasing evidence suggests that the reproducibility and replicability of scientific findings is threatened by researchers employing questionable research practices (QRP) in order to achieve publishable, positive and significant results. Numerous metrics have been developed to determine replication success but it has not yet been established how well those metrics perform in the presence of QRPs. This paper aims to compare the performance of different metrics quantifying replication success in the presence of four different types of QRPs: cherry picking, questionable interim analyses, questionable inclusion of covariates, and questionable subgroup analyses. Our results show that the metric based on the golden sceptical p -value does better in maintaining low values of overall type-I error rate, but often needs larger replication sample sizes, especially when severe QRPs are employed.

Suggested Citation

  • Freuli, Francesca & Held, Leonhard & Heyard, Rachel, 2022. "Replication success under questionable research practices – a simulation study," MetaArXiv s4b65, Center for Open Science.
    • Handle: RePEc:osf:metaar:s4b65
    DOI: 10.31219/osf.io/s4b65
    as

    Download full text from publisher

    File URL: https://osf.io/download/62ea3510136b5703f24545c3/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/s4b65?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Leonhard Held, 2020. "A new standard for the analysis and design of replication studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(2), pages 431-448, February.
    2. Wicherts, Jelte M. & Veldkamp, Coosje Lisabet Sterre & Augusteijn, Hilde & Bakker, Marjan & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2016. "Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking," OSF Preprints umq8d, Center for Open Science.
    3. Anne-Laure Boulesteix & Sabine Lauer & Manuel J A Eugster, 2013. "A Plea for Neutral Comparison Studies in Computational Sciences," PLOS ONE, Public Library of Science, vol. 8(4), pages 1-11, April.
    4. Larry V. Hedges & Jacob M. Schauer, 2019. "More Than One Replication Study Is Needed for Unambiguous Tests of Replication," Journal of Educational and Behavioral Statistics, , vol. 44(5), pages 543-570, October.
    5. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    6. Erik W. van Zwet & Eric A. Cator, 2021. "The significance filter, the winner's curse and the need to shrink," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 75(4), pages 437-452, November.
    7. Dorothy Bishop, 2019. "Rein in the four horsemen of irreproducibility," Nature, Nature, vol. 568(7753), pages 435-435, April.
    8. Wanja Wolff & Lorena Baumann & Chris Englert, 2018. "Self-reports from behind the scenes: Questionable research practices and rates of replication in ego depletion research," PLOS ONE, Public Library of Science, vol. 13(6), pages 1-11, June.
    9. Gowri Gopalakrishna & Gerben ter Riet & Gerko Vink & Ineke Stoop & Jelte M Wicherts & Lex M Bouter, 2022. "Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-16, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Samuel Pawel & Frederik Aust & Leonhard Held & Eric-Jan Wagenmakers, 2024. "Power priors for replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(1), pages 127-154, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:osf:metaar:s4b65_v1 is not listed on IDEAS
    2. Freuli, Francesca & Held, Leonhard & Heyard, Rachel, 2022. "Replication Success under Questionable Research Practices - A Simulation Study," I4R Discussion Paper Series 2, The Institute for Replication (I4R).
    3. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    4. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    5. Samuel Pawel & Leonhard Held, 2022. "The sceptical Bayes factor for the assessment of replication success," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(3), pages 879-911, July.
    6. Konstantinos Bourazas & Guido Consonni & Laura Deldossi, 2024. "Bayesian sample size determination for detecting heterogeneity in multi-site replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(3), pages 697-716, September.
    7. Cantone, Giulio Giacomo, 2023. "The multiversal methodology as a remedy of the replication crisis," MetaArXiv kuhmz, Center for Open Science.
    8. Heyard, Rachel & Pawel, Samuel & Frese, Joris & Voelkl, Bernhard & Würbel, Hanno & McCann, Sarah & Held, Leonhard & Wever, Kimberley E. PhD & Hartmann, Helena & Townsin, Louise, 2024. "A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics," MetaArXiv apdxk, Center for Open Science.
    9. repec:osf:metaar:kuhmz_v1 is not listed on IDEAS
    10. repec:osf:metaar:apdxk_v1 is not listed on IDEAS
    11. Samuel Pawel & Frederik Aust & Leonhard Held & Eric-Jan Wagenmakers, 2024. "Power priors for replication studies," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 33(1), pages 127-154, March.
    12. Craig, Russell & Cox, Adam & Tourish, Dennis & Thorpe, Alistair, 2020. "Using retracted journal articles in psychology to understand research misconduct in the social sciences: What is to be done?," Research Policy, Elsevier, vol. 49(4).
    13. Briony Swire-Thompson & David Lazer, 2022. "Reducing Health Misinformation in Science: A Call to Arms," The ANNALS of the American Academy of Political and Social Science, , vol. 700(1), pages 124-135, March.
    14. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    15. Mattia Prosperi & Jiang Bian & Iain E. Buchan & James S. Koopman & Matthew Sperrin & Mo Wang, 2019. "Raiders of the lost HARK: a reproducible inference framework for big data science," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    16. Huber, Christoph & Dreber, Anna & Huber, Jürgen & Johannesson, Magnus & Kirchler, Michael & Weitzel, Utz & Abellán, Miguel & Adayeva, Xeniya & Ay, Fehime Ceren & Barron, Kai & Berry, Zachariah & Bönte, 2023. "Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 120(23), pages 1-1.
    17. Charlotte Micheloud & Fadoua Balabdaoui & Leonhard Held, 2023. "Assessing replicability with the sceptical p$$ p $$‐value: Type‐I error control and sample size planning," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 77(4), pages 573-591, November.
    18. Deer, Lachlan & Adler, Susanne J. & Datta, Hannes & Mizik, Natalie & Sarstedt, Marko, 2025. "Toward open science in marketing research," International Journal of Research in Marketing, Elsevier, vol. 42(1), pages 212-233.
    19. Leonhard Held, 2020. "The harmonic mean χ2‐test to substantiate scientific findings," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(3), pages 697-708, June.
    20. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    21. Maximilian M Mandl & Sabine Hoffmann & Sebastian Bieringer & Anna E Jacob & Marie Kraft & Simon Lemster & Anne-Laure Boulesteix, 2024. "Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices," PLOS Computational Biology, Public Library of Science, vol. 20(3), pages 1-10, March.
    22. repec:osf:socarx:e26qf_v1 is not listed on IDEAS
    23. repec:osf:osfxxx:gmfs9_v1 is not listed on IDEAS
    24. Katharina Paulick & Simon Seidel & Christoph Lange & Annina Kemmer & Mariano Nicolas Cruz-Bournazou & André Baier & Daniel Haehn, 2022. "Promoting Sustainability through Next-Generation Biologics Drug Development," Sustainability, MDPI, vol. 14(8), pages 1-31, April.
    25. Marieke Stolte & Nicholas Schreck & Alla Slynko & Maral Saadati & Axel Benner & Jörg Rahnenführer & Andrea Bommert & for the topic group “High-dimensional data” (TG9) of the STRATOS Initiative, 2025. "Simulation study to evaluate when Plasmode simulation is superior to parametric simulation in comparing classification methods on high-dimensional data," PLOS ONE, Public Library of Science, vol. 20(6), pages 1-36, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:s4b65. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.