IDEAS home Printed from https://ideas.repec.org/p/fip/fedgfe/2019-16.html
   My bibliography  Save this paper

The Limits of p-Hacking : A Thought Experiment

Author

Listed:
  • Andrew Y. Chen

Abstract

Suppose that asset pricing factors are just p-hacked noise. How much p-hacking is required to produce the 300 factors documented by academics? I show that, if 10,000 academics generate 1 factor every minute, it takes 15 million years of p-hacking. This absurd conclusion comes from applying the p-hacking theory to published data. To fit the fat right tail of published t-stats, the p-hacking theory requires that the probability of publishing t-stats

Suggested Citation

  • Andrew Y. Chen, 2019. "The Limits of p-Hacking : A Thought Experiment," Finance and Economics Discussion Series 2019-016, Board of Governors of the Federal Reserve System (U.S.).
  • Handle: RePEc:fip:fedgfe:2019-16
    DOI: 10.17016/FEDS.2019.016
    as

    Download full text from publisher

    File URL: https://www.federalreserve.gov/econres/feds/files/2019016pap.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Mehra, Rajnish & Prescott, Edward C., 1985. "The equity premium: A puzzle," Journal of Monetary Economics, Elsevier, vol. 15(2), pages 145-161, March.
    2. Juhani T Linnainmaa & Michael R Roberts, 2018. "The History of the Cross-Section of Stock Returns," Review of Financial Studies, Society for Financial Studies, vol. 31(7), pages 2606-2649.
    3. Martin Lettau & Markus Pelger & Stijn Van Nieuwerburgh, 2020. "Factors That Fit the Time Series and Cross-Section of Stock Returns," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2274-2325.
    4. Kozak, Serhiy & Nagel, Stefan & Santosh, Shrihari, 2020. "Shrinking the cross-section," Journal of Financial Economics, Elsevier, vol. 135(2), pages 271-292.
    5. R. David Mclean & Jeffrey Pontiff, 2016. "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, American Finance Association, vol. 71(1), pages 5-32, February.
    6. Kewei Hou & Chen Xue & Lu Zhang, 2015. "Editor's Choice Digesting Anomalies: An Investment Approach," Review of Financial Studies, Society for Financial Studies, vol. 28(3), pages 650-705.
    7. Andrew Y Chen & Tom Zimmermann & Jeffrey Pontiff, 2020. "Publication Bias and the Cross-Section of Stock Returns," Review of Asset Pricing Studies, Oxford University Press, vol. 10(2), pages 249-289.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ruoxuan Xiong & Markus Pelger, 2019. "Large Dimensional Latent Factor Modeling with Missing Observations and Applications to Causal Inference," Papers 1910.08273, arXiv.org, revised Nov 2020.
    2. Andrei, Daniel & Cujean, Julien & Fournier, Mathieu, 2019. "The Low-Minus-High Portfolio and the Factor Zoo," CEPR Discussion Papers 14153, C.E.P.R. Discussion Papers.
    3. G Andrew Karolyi & Stijn Van Nieuwerburgh, 2020. "New Methods for the Cross-Section of Returns," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 1879-1890.
    4. Cederburg, Scott & O’Doherty, Michael S. & Wang, Feifei & Yan, Xuemin (Sterling), 2020. "On the performance of volatility-managed portfolios," Journal of Financial Economics, Elsevier, vol. 138(1), pages 95-117.
    5. Lettau, Martin & Pelger, Markus, 2020. "Estimating latent asset-pricing factors," Journal of Econometrics, Elsevier, vol. 218(1), pages 1-31.
    6. Matti Keloharju & Juhani T. Linnainmaa & Peter Nyberg, 2019. "Long-Term Discount Rates Do Not Vary Across Firms," NBER Working Papers 25579, National Bureau of Economic Research, Inc.
    7. Joachim Freyberger & Andreas Neuhierl & Michael Weber & Andrew Karolyi, 2020. "Dissecting Characteristics Nonparametrically," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2326-2377.
    8. Guanhao Feng & Stefano Giglio & Dacheng Xiu, 2020. "Taming the Factor Zoo: A Test of New Factors," Journal of Finance, American Finance Association, vol. 75(3), pages 1327-1370, June.
    9. Jacobs, Heiko & Müller, Sebastian, 2020. "Anomalies across the globe: Once public, no longer existent?," Journal of Financial Economics, Elsevier, vol. 135(1), pages 213-230.
    10. Alexander M. Chinco & Samuel M. Hartzmark & Abigail B. Sussman, 2020. "Necessary Evidence For A Risk Factor’s Relevance," NBER Working Papers 27227, National Bureau of Economic Research, Inc.
    11. Hoang, Khoa & Cannavan, Damien & Gaunt, Clive & Huang, Ronghong, 2019. "Is that factor just lucky? Australian evidence," Pacific-Basin Finance Journal, Elsevier, vol. 57(C).
    12. Croce, M.M. & Nguyen, Thien T. & Raymond, S. & Schmid, L., 2019. "Government debt and the returns to innovation," Journal of Financial Economics, Elsevier, vol. 132(3), pages 205-225.
    13. Long, Huaigang & Zhu, Yanjian & Chen, Lifang & Jiang, Yuexiang, 2019. "Tail risk and expected stock returns around the world," Pacific-Basin Finance Journal, Elsevier, vol. 56(C), pages 162-178.
    14. Pätäri, Eero & Karell, Ville & Luukka, Pasi & Yeomans, Julian S, 2018. "Comparison of the multicriteria decision-making methods for equity portfolio selection: The U.S. evidence," European Journal of Operational Research, Elsevier, vol. 265(2), pages 655-672.
    15. Adam Zaremba & Jacob Koby Shemer, 2018. "Price-Based Investment Strategies," Springer Books, Springer, number 978-3-319-91530-2, January.
    16. Kozak, Serhiy & Nagel, Stefan & Santosh, Shrihari, 2020. "Shrinking the cross-section," Journal of Financial Economics, Elsevier, vol. 135(2), pages 271-292.
    17. Geertsema, Paul & Lu, Helen, 2020. "The correlation structure of anomaly strategies," Journal of Banking & Finance, Elsevier, vol. 119(C).
    18. John H. Cochrane, 2017. "Macro-Finance," Review of Finance, European Finance Association, vol. 21(3), pages 945-985.
    19. Dittmar, Robert F. & Lundblad, Christian T., 2017. "Firm characteristics, consumption risk, and firm-level risk exposures," Journal of Financial Economics, Elsevier, vol. 125(2), pages 326-343.
    20. Yukun Liu & Aleh Tsyvinski, 2018. "Risks and Returns of Cryptocurrency," NBER Working Papers 24877, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    Stock return anomalies; Multiple testing; p-hacking; Publication bias;
    All these keywords.

    JEL classification:

    • G10 - Financial Economics - - General Financial Markets - - - General (includes Measurement and Data)
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedgfe:2019-16. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: http://edirc.repec.org/data/frbgvus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.