IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1906.06711.html
   My bibliography  Save this paper

Detecting p-hacking

Author

Listed:
  • Graham Elliott
  • Nikolay Kudrin
  • Kaspar Wuthrich

Abstract

We analyze what can be learned from tests for p-hacking based on distributions of t-statistics and p-values across multiple studies. We analytically characterize restrictions on these distributions that conform with the absence of p-hacking. This forms a testable null hypothesis and suggests statistical tests for p-hacking. We extend our results to p-hacking when there is also publication bias, and also consider what types of distributions arise under the alternative hypothesis that researchers engage in p-hacking. We show that the power of statistical tests for detecting p-hacking is low even if p-hacking is quite prevalent.

Suggested Citation

  • Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2019. "Detecting p-hacking," Papers 1906.06711, arXiv.org, revised Oct 2019.
    • Handle: RePEc:arx:papers:1906.06711
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1906.06711
    File Function: Latest version
    Download Restriction: no

    References listed on IDEAS

    as
    1. Gerber, Alan & Malhotra, Neil, 2008. "Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals," Quarterly Journal of Political Science, now publishers, vol. 3(3), pages 313-326, October.
    2. Joseph P. Romano & Michael Wolf, 2011. "Testing for monotonicity in expected asset returns," ECON - Working Papers 017, Department of Economics - University of Zurich, revised Jan 2013.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    5. repec:aea:aecrev:v:109:y:2019:i:8:p:2766-94 is not listed on IDEAS
    6. repec:bla:obuest:v:81:y:2019:i:4:p:797-816 is not listed on IDEAS
    7. Christopher A. Carolan & Joshua M. Tebbs, 2005. "Nonparametric tests for and against likelihood ratio ordering in the two-sample problem," Biometrika, Biometrika Trust, vol. 92(1), pages 159-171, March.
    8. Beare, Brendan K. & Moon, Jong-Myun, 2015. "Nonparametric Tests Of Density Ratio Ordering," Econometric Theory, Cambridge University Press, vol. 31(03), pages 471-492, June.
    9. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    10. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    11. repec:bla:jorssc:v:34:y:1985:i:3:p:320-325 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1906.06711. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (arXiv administrators). General contact details of provider: http://arxiv.org/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.