IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0306911.html
   My bibliography  Save this article

Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

Author

Listed:
  • Xingyu Li
  • Jiting Liu
  • Weijia Gao
  • Geoffrey L Cohen

Abstract

Large sample size (N) is seen as a key criterion in judging the replicability of psychological research, a phenomenon we refer to as the N-Heuristic. This heuristic has led to the incentivization of fast, online, non-behavioral studies—to the potential detriment of psychological science. While large N should in principle increase statistical power and thus the replicability of effects, in practice it may not. Large-N studies may have other attributes that undercut their power or validity. Consolidating data from all systematic, large-scale attempts at replication (N = 307 original-replication study pairs), we find that the original study’s sample size did not predict its likelihood of being replicated (rs = -0.02, p = 0.741), even with study design and research area controlled. By contrast, effect size emerged as a substantial predictor (rs = 0.21, p

Suggested Citation

  • Xingyu Li & Jiting Liu & Weijia Gao & Geoffrey L Cohen, 2024. "Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science," PLOS ONE, Public Library of Science, vol. 19(8), pages 1-15, August.
    • Handle: RePEc:plo:pone00:0306911
    DOI: 10.1371/journal.pone.0306911
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0306911
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0306911&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0306911?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Adam Altmejd & Anna Dreber & Eskil Forsell & Juergen Huber & Taisuke Imai & Magnus Johannesson & Michael Kirchler & Gideon Nave & Colin Camerer, 2019. "Predicting the replicability of social science lab experiments," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-18, December.
    2. Friedman, Lynn & Wall, Melanie, 2005. "Graphical Views of Suppression and Multicollinearity in Multiple Linear Regression," The American Statistician, American Statistical Association, vol. 59, pages 127-136, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexandru Marcoci & David P. Wilkinson & Ans Vercammen & Bonnie C. Wintle & Anna Lou Abatayo & Ernest Baskin & Henk Berkman & Erin M. Buchanan & Sara Capitán & Tabaré Capitán & Ginny Chan & Kent Jason, 2025. "Predicting the replicability of social and behavioural science claims in COVID-19 preprints," Nature Human Behaviour, Nature, vol. 9(2), pages 287-304, February.
    2. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    3. Chatelain, Jean-Bernard & Ralf, Kirsten, 2014. "Spurious regressions and near-multicollinearity, with an application to aid, policies and growth," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 39(A), pages 85-96.
    4. Dang,Hai-Anh H. & Lanjouw,Peter F., 2013. "Measuring poverty dynamics with synthetic panels based on cross-sections," Policy Research Working Paper Series 6504, The World Bank.
    5. Felix Holzmeister & Magnus Johannesson & Colin F. Camerer & Yiling Chen & Teck-Hua Ho & Suzanne Hoogeveen & Juergen Huber & Noriko Imai & Taisuke Imai & Lawrence Jin & Michael Kirchler & Alexander Ly , 2025. "Examining the replicability of online experiments selected by a decision market," Nature Human Behaviour, Nature, vol. 9(2), pages 316-330, February.
    6. Aizenkot, Dana, 2020. "Social networking and online self-disclosure as predictors of cyberbullying victimization among children and youth," Children and Youth Services Review, Elsevier, vol. 119(C).
    7. Gijung Jung & Jia Lee, 2022. "Behavioral and Psychological Symptoms and Associated Factors in Community-Dwelling Persons at the First Time of Dementia Diagnosis," IJERPH, MDPI, vol. 19(13), pages 1-11, June.
    8. Jens Rommel & Meike Weltin, 2021. "Is There a Cult of Statistical Significance in Agricultural Economics?," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(3), pages 1176-1191, September.
    9. repec:osf:metaar:e9nw2_v1 is not listed on IDEAS
    10. Diya Dou & Daniel T. L. Shek & Xiaoqin Zhu & Li Zhao, 2021. "Dimensionality of the Chinese CES-D: Is It Stable across Gender, Time, and Samples?," IJERPH, MDPI, vol. 18(22), pages 1-11, November.
    11. Isager, Peder Mortvedt & van 't Veer, Anna Elisabeth & Lakens, Daniel, 2021. "Replication value as a function of citation impact and sample size," MetaArXiv knjea, Center for Open Science.
    12. Niels Waller, 2011. "The Geometry of Enhancement in Multiple Regression," Psychometrika, Springer;The Psychometric Society, vol. 76(4), pages 634-649, October.
    13. Nazifi Morteza & Fadishei Hamid, 2022. "Supsim: a Python package and a web-based JavaScript tool to address the theoretical complexities in two-predictor suppression situations," Statistics in Transition New Series, Statistics Poland, vol. 23(4), pages 177-202, December.
    14. Vu, Patrick, 2024. "Why are replication rates so low?," Journal of Econometrics, Elsevier, vol. 245(1).
    15. Richard Briesch & Ernan Haruvy & Glenn B. Voss & Zannie Giraud Voss, 2024. "The countervailing effects of spatial competition in the performing arts: examining local versus traded market performance," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 48(4), pages 527-574, December.
    16. Peter Hendrix & Michael Ramscar & Harald Baayen, 2019. "NDRA: A single route model of response times in the reading aloud task based on discriminative learning," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-63, July.
    17. Duncan Ermini Leaf, 2023. "Risk management in the use of published statistical results for policy decisions," Papers 2305.03205, arXiv.org, revised Aug 2024.
    18. Alipourfard, Nazanin & Arendt, Beatrix & Benjamin, Daniel Jacob & Benkler, Noam & Bishop, Michael Metcalf & Burstein, Mark & Bush, Martin & Caverlee, James & Chen, Yiling & Clark, Chae, 2021. "Systematizing Confidence in Open Research and Evidence (SCORE)," SocArXiv 46mnb, Center for Open Science.
    19. Adler, Susanne Jana & Röseler, Lukas & Schöniger, Martina Katharina, 2023. "A toolbox to evaluate the trustworthiness of published findings," Journal of Business Research, Elsevier, vol. 167(C).
    20. Franke, George R. & Nadler, S. Scott, 2008. "Culture, economic development, and national ethical attitudes," Journal of Business Research, Elsevier, vol. 61(3), pages 254-264, March.
    21. Suzanne V. Landram & Frank G. Landram, 2012. "A computational understanding of partial and part determination coefficients," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(12), pages 2619-2626, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0306911. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.