IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1011936.html
   My bibliography  Save this article

Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices

Author

Listed:
  • Maximilian M Mandl
  • Sabine Hoffmann
  • Sebastian Bieringer
  • Anna E Jacob
  • Marie Kraft
  • Simon Lemster
  • Anne-Laure Boulesteix

Abstract

Author summary: Throughout their education and when reading the scientific literature, students may get the impression that there is a unique and correct analysis strategy for every data analysis task and that this analysis strategy will always yield a significant and noteworthy result. This expectation conflicts with a growing realization that there is a multiplicity of possible analysis strategies in empirical research, which will lead to overoptimism and nonreplicable research findings if it is combined with result-dependent selective reporting. Here, we argue that students are often ill-equipped for real-world data analysis tasks and unprepared for the dangers of selectively reporting the most promising results. We present a seminar course intended for advanced undergraduates and beginning graduate students of data analysis fields such as statistics, data science, or bioinformatics that aims to increase the awareness of uncertain choices in the analysis of empirical data and present ways to deal with these choices through theoretical modules and practical hands-on sessions.

Suggested Citation

  • Maximilian M Mandl & Sabine Hoffmann & Sebastian Bieringer & Anna E Jacob & Marie Kraft & Simon Lemster & Anne-Laure Boulesteix, 2024. "Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices," PLOS Computational Biology, Public Library of Science, vol. 20(3), pages 1-10, March.
    • Handle: RePEc:plo:pcbi00:1011936
    DOI: 10.1371/journal.pcbi.1011936
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011936
    Download Restriction: no
    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1011936&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pcbi.1011936?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Anne-Laure Boulesteix, 2015. "Ten Simple Rules for Reducing Overoptimistic Reporting in Methodological Computational Research," PLOS Computational Biology, Public Library of Science, vol. 11(4), pages 1-6, April.
    2. repec:plo:pcbi00:1003285 is not listed on IDEAS
    3. repec:osf:osfxxx:umq8d_v1 is not listed on IDEAS
    4. Wicherts, Jelte M. & Veldkamp, Coosje Lisabet Sterre & Augusteijn, Hilde & Bakker, Marjan & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2016. "Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking," OSF Preprints umq8d, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    2. repec:osf:metaar:s4b65_v1 is not listed on IDEAS
    3. Silke Janitza & Ender Celik & Anne-Laure Boulesteix, 2018. "A computationally fast variable importance test for random forests for high-dimensional data," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(4), pages 885-915, December.
    4. repec:osf:osfxxx:gv25c_v1 is not listed on IDEAS
    5. Theresa Ullmann & Anna Beer & Maximilian Hünemörder & Thomas Seidl & Anne-Laure Boulesteix, 2023. "Over-optimistic evaluation and reporting of novel cluster algorithms: an illustrative study," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(1), pages 211-238, March.
    6. repec:osf:osfxxx:7dc6a_v1 is not listed on IDEAS
    7. repec:osf:metaar:2bj85_v1 is not listed on IDEAS
    8. Alexander Mielke, 2023. "Impact of dominance rank specification in dyadic interaction models," PLOS ONE, Public Library of Science, vol. 18(7), pages 1-19, July.
    9. Furlong, Aidan & Alsafadi, Farah & Palmtag, Scott & Godfrey, Andrew & Wu, Xu, 2025. "Data-driven prediction and uncertainty quantification of PWR crud-induced power shift using convolutional neural networks," Energy, Elsevier, vol. 316(C).
    10. Freuli, Francesca & Held, Leonhard & Heyard, Rachel, 2022. "Replication Success under Questionable Research Practices - A Simulation Study," I4R Discussion Paper Series 2, The Institute for Replication (I4R).
    11. Daniel McNeish, 2024. "Practical Implications of Sum Scores Being Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1148-1169, December.
    12. Thibaut Arpinon & Romain Espinosa, 2023. "A Practical Guide to Registered Reports for Economists," Post-Print halshs-03897719, HAL.
    13. Lisa Spitzer & Stefanie Mueller, 2023. "Registered report: Survey on attitudes and experiences regarding preregistration in psychological research," PLOS ONE, Public Library of Science, vol. 18(3), pages 1-34, March.
    14. Shaw, Steven D. & Nave, Gideon, 2023. "Don't hate the player, hate the game: Realigning incentive structures to promote robust science and better scientific practices in marketing," Journal of Business Research, Elsevier, vol. 167(C).
    15. Gabriel Nova & Sander van Cranenburgh & Stephane Hess, 2024. "Understanding the decision-making process of choice modellers," Papers 2411.01704, arXiv.org, revised Jun 2025.
    16. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    17. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    18. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArchive 2b5k4_v1, Center for Open Science.
    19. Denise Rousseau & Byeong Jo Kim & Ryan Splenda & Sarah Young & Jangbum Lee & Donna Beck, 2023. "Does chief executive compensation predict financial performance or inaccurate financial reporting in listed companies: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 19(4), December.
    20. Qingzhen Hou & Katharina Waury & Dea Gogishvili & K Anton Feenstra, 2022. "Ten quick tips for sequence-based prediction of protein properties using machine learning," PLOS Computational Biology, Public Library of Science, vol. 18(12), pages 1-15, December.
    21. repec:osf:osfxxx:ksxwv_v1 is not listed on IDEAS
    22. Christian Hennig, 2022. "An empirical comparison and characterisation of nine popular clustering methods," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 16(1), pages 201-229, March.
    23. Augusteijn, Hilde Elisabeth Maria & Wicherts, Jelte M. & Sijtsma, Klaas & van Assen, Marcel A. L. M., 2023. "Quality assessment of scientific manuscripts in peer review and education," OSF Preprints 7dc6a, Center for Open Science.
    24. Freuli, Francesca & Held, Leonhard & Heyard, Rachel, 2022. "Replication success under questionable research practices – a simulation study," MetaArXiv s4b65, Center for Open Science.
    25. Adler, Susanne Jana & Röseler, Lukas & Schöniger, Martina Katharina, 2023. "A toolbox to evaluate the trustworthiness of published findings," Journal of Business Research, Elsevier, vol. 167(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1011936. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.