IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0335330.html

Metascientific replication project with the advanced meta-experimental protocol of the transparent psi project procedures for testing the precognitive effect claimed by Bem

Author

Listed:
  • Jan Walleczek
  • Nikolaus von Stillfried
  • Stefan Schmidt
  • Marc Wittmann
  • Karolina A Kirmse
  • Jorge Moll
  • Zoltan Kekecs

Abstract

This metascientific project studied the replicability of Bem Experiment 1, which had claimed a precognitive effect, i.e., the ability to successfully guess the outcome of future random events (Bem. J Pers Soc Psychol. 2011;100: 407−25). The use of advanced methodologies—based on the advanced meta-experimental protocol (AMP) and transparent psi project (TPP) procedures—reduced the risk of false discoveries as a function of (i) confirmation bias, (ii) non-transparency, and (iii) intrinsic measurement bias. The combined AMP-TPP test strategy performed three replication studies with a total of 26,483 participants resulting in N = 420,472 critical trials. Study 1 failed to replicate the precognitive effect. An exploratory analysis of Study 1 suggested an effect in the opposite direction than was originally predicted (49.48% ± 0.26 SE; N = 37,836). Study 2 confirmed this exploratory result using a high-powered replication design (49.65% ± 0.14 SE; p = 0.013; N = 127,000). Study 3 was unable to replicate the result from Study 2 (50.07% ± 0.11 SE; p = 0.496; N = 217,800). The results of Study 2 represent a rare example in psi research of the successful replication of an exploratory result using a confirmatory protocol. The source of the one-time confirmed anomalous result in Study 2 remains to be identified. This result presents either (i) a psi-derived anomaly that defies known physical laws, or (ii) a method-derived anomaly, e.g., a false-positive statistical finding. Using conventional standards, based on the lack of replicability in Study 3 and absence of an accepted scientific theory, the second scenario appears more plausible. This AMP-TPP metascientific project demonstrated the use of advanced controls for assessing the reliability of the employed scientific process. This project shows how rigor-enhancing test strategies can improve the reliability, not only of psi research, but any type of weak-effects experiments, including in psychology.

Suggested Citation

  • Jan Walleczek & Nikolaus von Stillfried & Stefan Schmidt & Marc Wittmann & Karolina A Kirmse & Jorge Moll & Zoltan Kekecs, 2025. "Metascientific replication project with the advanced meta-experimental protocol of the transparent psi project procedures for testing the precognitive effect claimed by Bem," PLOS ONE, Public Library of Science, vol. 20(11), pages 1-31, November.
    • Handle: RePEc:plo:pone00:0335330
    DOI: 10.1371/journal.pone.0335330
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0335330
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0335330&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0335330?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Dick J Bierman & James P Spottiswoode & Aron Bijl, 2016. "Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-18, May.
    2. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    3. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    4. repec:osf:osfxxx:2dxu5_v1 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Deer, Lachlan & Adler, Susanne J. & Datta, Hannes & Mizik, Natalie & Sarstedt, Marko, 2025. "Toward open science in marketing research," International Journal of Research in Marketing, Elsevier, vol. 42(1), pages 212-233.
    2. repec:osf:metaar:u4g6t_v1 is not listed on IDEAS
    3. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    4. Christopher Allen & David M A Mehler, 2019. "Open science challenges, benefits and tips in early career and beyond," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-14, May.
    5. Mattia Prosperi & Jiang Bian & Iain E. Buchan & James S. Koopman & Matthew Sperrin & Mo Wang, 2019. "Raiders of the lost HARK: a reproducible inference framework for big data science," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    6. Kraft-Todd, Gordon T. & Rand, David G., 2021. "Practice what you preach: Credibility-enhancing displays and the growth of open science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 164(C), pages 1-10.
    7. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawRxiv 2b5k4, Center for Open Science.
    8. Brinkerink, Jasper & De Massis, Alfredo & Kellermanns, Franz, 2022. "One finding is no finding: Toward a replication culture in family business research," Journal of Family Business Strategy, Elsevier, vol. 13(4).
    9. Filip Melinscak & Dominik R Bach, 2020. "Computational optimization of associative learning experiments," PLOS Computational Biology, Public Library of Science, vol. 16(1), pages 1-23, January.
    10. Nosek, Brian A. & Errington, Timothy M., 2019. "What is replication?," MetaArXiv u4g6t, Center for Open Science.
    11. Merton S. Krause, 2019. "Replication and preregistration," Quality & Quantity: International Journal of Methodology, Springer, vol. 53(5), pages 2647-2652, September.
    12. Logg, Jennifer M. & Dorison, Charles A., 2021. "Pre-registration: Weighing costs and benefits for researchers," Organizational Behavior and Human Decision Processes, Elsevier, vol. 167(C), pages 18-27.
    13. Persson, Emil & Tinghög, Gustav, 2020. "Opportunity cost neglect in public policy," Journal of Economic Behavior & Organization, Elsevier, vol. 170(C), pages 301-312.
    14. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    15. Vigren, Andreas & Pyddoke, Roger, 2020. "The impact on bus ridership of passenger incentive contracts in public transport," Transportation Research Part A: Policy and Practice, Elsevier, vol. 135(C), pages 144-159.
    16. Caicedo, Juan D. & Guirado, Carlos & González, Marta C. & Walker, Joan L., 2025. "Sharing, collaborating, and benchmarking to advance travel demand research: A demonstration of short-term ridership prediction," Transport Policy, Elsevier, vol. 171(C), pages 531-541.
    17. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    18. Hensel, Przemysław G., 2019. "Supporting replication research in management journals: Qualitative analysis of editorials published between 1970 and 2015," European Management Journal, Elsevier, vol. 37(1), pages 45-57.
    19. Severinsen, A. & Myrland, Ø., 2022. "ShinyRBase: Near real-time energy saving models using reactive programming," Applied Energy, Elsevier, vol. 325(C).
    20. Shane Timmons & Terence J. McElvaney & Peter D. Lunn, 2019. "An experiment for regulatory policy on broadband speed advertising," Journal of Behavioral Economics for Policy, Society for the Advancement of Behavioral Economics (SABE), vol. 3(2), pages 17-24, December.
    21. Elbæk, Christian T. & Lystbæk, Martin Nørhede & Mitkidis, Panagiotis, 2022. "On the psychology of bonuses: The effects of loss aversion and Yerkes-Dodson law on performance in cognitively and mechanically demanding tasks," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0335330. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.