IDEAS home Printed from https://ideas.repec.org/a/bla/jfinan/v76y2021i5p2447-2480.html
   My bibliography  Save this article

The Limits of p‐Hacking: Some Thought Experiments

Author

Listed:
  • ANDREW Y. CHEN

Abstract

Suppose that the 300+ published asset pricing factors are all spurious. How much p‐hacking is required to produce these factors? If 10,000 researchers generate eight factors every day, it takes hundreds of years. This is because dozens of published t‐statistics exceed 6.0, while the corresponding p‐value is infinitesimal, implying an astronomical amount of p‐hacking in a general model. More structure implies that p‐hacking cannot address ≈100 published t‐statistics that exceed 4.0, as they require an implausibly nonlinear preference for t‐statistics or even more p‐hacking. These results imply that mispricing, risk, and/or frictions have a key role in stock returns.

Suggested Citation

  • Andrew Y. Chen, 2021. "The Limits of p‐Hacking: Some Thought Experiments," Journal of Finance, American Finance Association, vol. 76(5), pages 2447-2480, October.
    • Handle: RePEc:bla:jfinan:v:76:y:2021:i:5:p:2447-2480
    DOI: 10.1111/jofi.13036
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/jofi.13036
    Download Restriction: no
    File URL: https://libkey.io/10.1111/jofi.13036?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kelly, Bryan T. & Pruitt, Seth & Su, Yinan, 2019. "Characteristics are covariances: A unified model of risk and return," Journal of Financial Economics, Elsevier, vol. 134(3), pages 501-524.
    2. Weitzman, Martin L, 1979. "Optimal Search for the Best Alternative," Econometrica, Econometric Society, vol. 47(3), pages 641-654, May.
    3. Tarun Chordia & Amit Goyal & Alessio Saretto & Andrew KarolyiEditor, 2020. "Anomalies and False Rejections," Review of Finance, European Finance Association, vol. 33(5), pages 2134-2179.
    4. Andrew Y. Chen & Tom Zimmermann, 2022. "Open Source Cross-Sectional Asset Pricing," Critical Finance Review, now publishers, vol. 11(2), pages 207-264, May.
    5. R. David Mclean & Jeffrey Pontiff, 2016. "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, American Finance Association, vol. 71(1), pages 5-32, February.
    6. Lewellen, Jonathan & Nagel, Stefan & Shanken, Jay, 2010. "A skeptical appraisal of asset pricing tests," Journal of Financial Economics, Elsevier, vol. 96(2), pages 175-194, May.
    7. Kewei Hou & Chen Xue & Lu Zhang, 2015. "Editor's Choice Digesting Anomalies: An Investment Approach," The Review of Financial Studies, Society for Financial Studies, vol. 28(3), pages 650-705.
    8. Guanhao Feng & Stefano Giglio & Dacheng Xiu, 2020. "Taming the Factor Zoo: A Test of New Factors," Journal of Finance, American Finance Association, vol. 75(3), pages 1327-1370, June.
    9. Joel E. Cohen, 2019. "Sum of a Random Number of Correlated Random Variables that Depend on the Number of Summands," The American Statistician, Taylor & Francis Journals, vol. 73(1), pages 56-60, January.
    10. Raymond Kan & Chu Zhang, 1999. "Two‐Pass Tests of Asset Pricing Models with Useless Factors," Journal of Finance, American Finance Association, vol. 54(1), pages 203-235, February.
    11. Train,Kenneth E., 2009. "Discrete Choice Methods with Simulation," Cambridge Books, Cambridge University Press, number 9780521766555, Enero.
    12. Fama, Eugene F. & French, Kenneth R., 2015. "A five-factor asset pricing model," Journal of Financial Economics, Elsevier, vol. 116(1), pages 1-22.
    13. Ryan Sullivan & Allan Timmermann & Halbert White, 1999. "Data‐Snooping, Technical Trading Rule Performance, and the Bootstrap," Journal of Finance, American Finance Association, vol. 54(5), pages 1647-1691, October.
    14. Kenneth Burdett & Shouyong Shi & Randall Wright, 2001. "Pricing and Matching with Frictions," Journal of Political Economy, University of Chicago Press, vol. 109(5), pages 1060-1085, October.
    15. Martin Lettau & Markus Pelger & Stijn Van Nieuwerburgh, 2020. "Factors That Fit the Time Series and Cross-Section of Stock Returns," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2274-2325.
    16. Lo, Andrew W & MacKinlay, A Craig, 1990. "Data-Snooping Biases in Tests of Financial Asset Pricing Models," The Review of Financial Studies, Society for Financial Studies, vol. 3(3), pages 431-467.
    17. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    18. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Papers 2006.04269, arXiv.org.
    19. Andrew Y Chen & Tom Zimmermann & Jeffrey Pontiff, 2020. "Publication Bias and the Cross-Section of Stock Returns," The Review of Asset Pricing Studies, Society for Financial Studies, vol. 10(2), pages 249-289.
    20. Nicholas C. Barberis, 2018. "Psychology-based Models of Asset Prices and Trading Volume," NBER Working Papers 24723, National Bureau of Economic Research, Inc.
    21. Glenn Ellison, 2002. "Evolving Standards for Academic Publishing: A q-r Theory," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 994-1034, October.
    22. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    23. Jacobs, Heiko & Müller, Sebastian, 2020. "Anomalies across the globe: Once public, no longer existent?," Journal of Financial Economics, Elsevier, vol. 135(1), pages 213-230.
    24. Serhiy Kozak & Stefan Nagel & Shrihari Santosh, 2018. "Interpreting Factor Models," Journal of Finance, American Finance Association, vol. 73(3), pages 1183-1223, June.
    25. Adam, Klaus, 2001. "Learning While Searching for the Best Alternative," Journal of Economic Theory, Elsevier, vol. 101(1), pages 252-280, November.
    26. John D. Storey & Jonathan E. Taylor & David Siegmund, 2004. "Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 66(1), pages 187-205, February.
    27. Efron B. & Tibshirani R. & Storey J.D. & Tusher V., 2001. "Empirical Bayes Analysis of a Microarray Experiment," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1151-1160, December.
    28. Tarun Chordia & Amit Goyal & Alessio Saretto, 2020. "Anomalies and False Rejections," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2134-2179.
    29. Campbell R. Harvey & Yan Liu & Heqing Zhu, 2016. "Editor's Choice … and the Cross-Section of Expected Returns," The Review of Financial Studies, Society for Financial Studies, vol. 29(1), pages 5-68.
    30. James D. Montgomery, 1991. "Equilibrium Wage Dispersion and Interindustry Wage Differentials," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 106(1), pages 163-179.
    31. Jeremiah Green & John R. M. Hand & X. Frank Zhang, 2017. "The Characteristics that Provide Independent Information about Average U.S. Monthly Stock Returns," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4389-4436.
    32. Yoav Benjamini, 2010. "Discovering the false discovery rate," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(4), pages 405-416, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andrew Y. Chen, 2022. "Most claimed statistical findings in cross-sectional return predictability are likely true," Papers 2206.15365, arXiv.org, revised Jun 2025.
    2. Han, Yufeng & Lu, Yueliang (Jacques) & Xu, Weike & Zhou, Guofu, 2024. "Mispricing and Anomalies: An Exogenous Shock to Short Selling from JGTRRA," Journal of Empirical Finance, Elsevier, vol. 78(C).
    3. Andrew Y. Chen, 2022. "Do t-Statistic Hurdles Need to be Raised?," Papers 2204.10275, arXiv.org, revised Apr 2024.
    4. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    5. Venky Nagar & Jordan Schoenfeld, 2024. "Measuring weather exposure with annual reports," Review of Accounting Studies, Springer, vol. 29(1), pages 1-32, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Harvey, Campbell R. & Liu, Yan, 2021. "Lucky factors," Journal of Financial Economics, Elsevier, vol. 141(2), pages 413-435.
    2. Chinco, Alex & Neuhierl, Andreas & Weber, Michael, 2021. "Estimating the anomaly base rate," Journal of Financial Economics, Elsevier, vol. 140(1), pages 101-126.
    3. Söhnke M. Bartram & Harald Lohre & Peter F. Pope & Ananthalakshmi Ranganathan, 2021. "Navigating the factor zoo around the world: an institutional investor perspective," Journal of Business Economics, Springer, vol. 91(5), pages 655-703, July.
    4. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    5. Andrew Y. Chen, 2022. "Most claimed statistical findings in cross-sectional return predictability are likely true," Papers 2206.15365, arXiv.org, revised Jun 2025.
    6. Cakici, Nusret & Zaremba, Adam & Bianchi, Robert J. & Pham, Nga, 2021. "False discoveries in the anomaly research: New insights from the Stock Exchange of Melbourne (1927–1987)," Pacific-Basin Finance Journal, Elsevier, vol. 70(C).
    7. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    8. Wang, Jinzhe & Zhu, Yifeng, 2024. "A comparison of factor models in China," Journal of Empirical Finance, Elsevier, vol. 79(C).
    9. Kozak, Serhiy & Nagel, Stefan & Santosh, Shrihari, 2020. "Shrinking the cross-section," Journal of Financial Economics, Elsevier, vol. 135(2), pages 271-292.
    10. Andrew Y. Chen & Alejandro Lopez-Lira & Tom Zimmermann, 2022. "Does Peer-Reviewed Research Help Predict Stock Returns?," Papers 2212.10317, arXiv.org, revised Mar 2025.
    11. Cujean, Julien & Andrei, Daniel & Fournier, Mathieu, 2019. "The Low-Minus-High Portfolio and the Factor Zoo," CEPR Discussion Papers 14153, C.E.P.R. Discussion Papers.
    12. Hoang, Khoa & Huang, Ronghong & Truong, Helen, 2023. "Resurrecting the market factor: A case of data mining across international markets," Pacific-Basin Finance Journal, Elsevier, vol. 82(C).
    13. Lin William Cong & Guanhao Feng & Jingyu He & Xin He, 2022. "Growing the Efficient Frontier on Panel Trees," NBER Working Papers 30805, National Bureau of Economic Research, Inc.
    14. Gagliardini, Patrick & Ossola, Elisa & Scaillet, Olivier, 2020. "Estimation of large dimensional conditional factor models in finance," Handbook of Econometrics, in: Steven N. Durlauf & Lars Peter Hansen & James J. Heckman & Rosa L. Matzkin (ed.), Handbook of Econometrics, edition 1, volume 7, chapter 0, pages 219-282, Elsevier.
    15. Andrew Y. Chen, 2022. "Do t-Statistic Hurdles Need to be Raised?," Papers 2204.10275, arXiv.org, revised Apr 2024.
    16. Alex Chinco & Samuel M. Hartzmark & Abigail B. Sussman, 2022. "A New Test of Risk Factor Relevance," Journal of Finance, American Finance Association, vol. 77(4), pages 2183-2238, August.
    17. Assoe, Kodjovi & Attig, Najah & Sy, Oumar, 2024. "The battle of factors," Global Finance Journal, Elsevier, vol. 62(C).
    18. Vincent, Kendro & Hsu, Yu-Chin & Lin, Hsiou-Wei, 2021. "Investment styles and the multiple testing of cross-sectional stock return predictability," Journal of Financial Markets, Elsevier, vol. 56(C).
    19. Chen, Andrew Y. & McCoy, Jack, 2024. "Missing values handling for machine learning portfolios," Journal of Financial Economics, Elsevier, vol. 155(C).
    20. Gagliardini, Patrick & Ossola, Elisa & Scaillet, Olivier, 2019. "A diagnostic criterion for approximate factor structure," Journal of Econometrics, Elsevier, vol. 212(2), pages 503-521.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jfinan:v:76:y:2021:i:5:p:2447-2480. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/afaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.