IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v240y2024i1s030440762400023x.html
   My bibliography  Save this article

Classical p-values and the Bayesian posterior probability that the hypothesis is approximately true

Author

Listed:
  • Kline, Brendan

Abstract

This paper relates p-values for the hypothesis that θ=c to the Bayesian posterior probability that the hypothesis is approximately true, in the sense that θ∈[c−ϵ,c+ϵ] for a selected ϵ>0. In a setup with a continuous prior for θ, the results show that a larger (respectively, smaller) p-value does not necessarily correspond to a higher (respectively, lower) probability that θ is close to c. Therefore, the results suggest caution about common ways of using p-values, specifically the use of small p-values as a key standard in empirical research.

Suggested Citation

  • Kline, Brendan, 2024. "Classical p-values and the Bayesian posterior probability that the hypothesis is approximately true," Journal of Econometrics, Elsevier, vol. 240(1).
    • Handle: RePEc:eee:econom:v:240:y:2024:i:1:s030440762400023x
    DOI: 10.1016/j.jeconom.2024.105677
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S030440762400023X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jeconom.2024.105677?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kitagawa, Toru & Montiel Olea, José Luis & Payne, Jonathan & Velez, Amilcar, 2020. "Posterior distribution of nondifferentiable functions," Journal of Econometrics, Elsevier, vol. 217(1), pages 161-175.
    2. Alberto Abadie, 2020. "Statistical Nonsignificance in Empirical Economics," American Economic Review: Insights, American Economic Association, vol. 2(2), pages 193-208, June.
    3. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    4. Siddhartha Chib & Minchul Shin & Anna Simoni, 2018. "Bayesian Estimation and Comparison of Moment Condition Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1656-1668, October.
    5. Chernozhukov, Victor & Hong, Han, 2003. "An MCMC approach to classical estimation," Journal of Econometrics, Elsevier, vol. 115(2), pages 293-346, August.
    6. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    7. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    8. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    9. Brendan Kline, 2022. "Bayes Factors Based on p-Values and Sets of Priors With Restricted Strength," The American Statistician, Taylor & Francis Journals, vol. 76(3), pages 203-213, July.
    10. Liao, Yuan & Simoni, Anna, 2019. "Bayesian inference for partially identified smooth convex models," Journal of Econometrics, Elsevier, vol. 211(2), pages 338-360.
    11. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    12. Tom E. Hardwicke & Maia Salholz-Hillel & Mario Malički & Dénes Szűcs & Theiss Bendixen & John P. A. Ioannidis, 2023. "Statistical Guidance to Authors at Top-Ranked Journals across Scientific Disciplines," The American Statistician, Taylor & Francis Journals, vol. 77(3), pages 239-247, July.
    13. Liu, Xiaobin & Li, Yong & Yu, Jun & Zeng, Tao, 2022. "Posterior-based Wald-type statistics for hypothesis testing," Journal of Econometrics, Elsevier, vol. 230(1), pages 83-113.
    14. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    15. Xiaohong Chen & Timothy M. Christensen & Elie Tamer, 2018. "Monte Carlo Confidence Sets for Identified Sets," Econometrica, Econometric Society, vol. 86(6), pages 1965-2018, November.
    16. Leonhard Held & Manuela Ott, 2016. "How the Maximal Evidence of -Values Against Point Null Hypotheses Depends on Sample Size," The American Statistician, Taylor & Francis Journals, vol. 70(4), pages 335-341, October.
    17. Norets, Andriy, 2015. "Bayesian regression with nonparametric heteroskedasticity," Journal of Econometrics, Elsevier, vol. 185(2), pages 409-419.
    18. Kim, Jae-Young, 2002. "Limited information likelihood and Bayesian analysis," Journal of Econometrics, Elsevier, vol. 107(1-2), pages 175-193, March.
    19. Brendan Kline & Elie Tamer, 2016. "Bayesian inference in a class of partially identified models," Quantitative Economics, Econometric Society, vol. 7(2), pages 329-366, July.
    20. Athey, Susan & Imbens, Guido W., 2022. "Design-based analysis in Difference-In-Differences settings with staggered adoption," Journal of Econometrics, Elsevier, vol. 226(1), pages 62-79.
    21. repec:cup:polals:v:26:y:2018:i:01:p:1-2_00 is not listed on IDEAS
    22. Hyungsik Roger Moon & Frank Schorfheide, 2012. "Bayesian and Frequentist Inference in Partially Identified Models," Econometrica, Econometric Society, vol. 80(2), pages 755-782, March.
    23. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    24. Gafarov, Bulat & Meier, Matthias & Montiel Olea, José Luis, 2018. "Delta-method inference for a class of set-identified SVARs," Journal of Econometrics, Elsevier, vol. 203(2), pages 316-327.
    25. Valentin Amrhein & Sander Greenland & Blake McShane, 2019. "Scientists rise up against statistical significance," Nature, Nature, vol. 567(7748), pages 305-307, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sho Miyaji, 2024. "Instrumented Difference-in-Differences with Heterogeneous Treatment Effects," Papers 2405.12083, arXiv.org, revised Jul 2024.
    2. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    3. Emanuele Bacchiocchi & Toru Kitagawa, 2020. "Locally- but not globally-identified SVARs," CeMMAP working papers CWP40/20, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    4. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    5. Sho Miyaji, 2024. "Instrumented Difference-in-Differences with Heterogeneous Treatment Effects," Discussion Paper Series DP2024-22, Research Institute for Economics & Business Administration, Kobe University.
    6. Mac Clay, Pablo & Börner, Jan & Sellare, Jorge, 2023. "Institutional and macroeconomic stability mediate the effect of auctions on renewable energy capacity," Energy Policy, Elsevier, vol. 180(C).
    7. Rik Chakraborti & Gavin Roberts, 2023. "How price-gouging regulation undermined COVID-19 mitigation: county-level evidence of unintended consequences," Public Choice, Springer, vol. 196(1), pages 51-83, July.
    8. Han, Sukjin, 2021. "Identification in nonparametric models for dynamic treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 132-147.
    9. Jean-Pierre Florens & Anna Simoni, 2021. "Revisiting Identification Concepts in Bayesian Analysis," Annals of Economics and Statistics, GENES, issue 144, pages 1-38.
    10. Xin Nie & Jianxian Wu & Han Wang & Weijuan Li & Chengdao Huang & Lihua Li, 2022. "Contributing to carbon peak: Estimating the causal impact of eco‐industrial parks on low‐carbon development in China," Journal of Industrial Ecology, Yale University, vol. 26(4), pages 1578-1593, August.
    11. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Oct 2024.
    12. Cl'ement de Chaisemartin & Xavier D'Haultfoeuille, 2020. "Difference-in-Differences Estimators of Intertemporal Treatment Effects," Papers 2007.04267, arXiv.org, revised Nov 2023.
    13. Raffaella Giacomini & Toru Kitagawa, 2021. "Robust Bayesian Inference for Set‐Identified Models," Econometrica, Econometric Society, vol. 89(4), pages 1519-1556, July.
    14. Francesca Molinari, 2020. "Microeconometrics with Partial Identi?cation," CeMMAP working papers CWP15/20, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    15. Kurt Schmidheiny & Sebastian Siegloch, 2023. "On event studies and distributed‐lags in two‐way fixed effects models: Identification, equivalence, and generalization," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 695-713, August.
    16. Bhattacharjee, Arnab & Aravena, Claudia & Castillo, Natalia & Ehrlich, Marco & Taou, Nadia & Wagner, Thomas, 2022. "Agroforestry Programs in the Colombian Amazon: Selection, Treatment and Exposure Effects on Deforestation," National Institute of Economic and Social Research (NIESR) Discussion Papers 537, National Institute of Economic and Social Research.
    17. Kayaoglu, Aysegul, 2022. "Do refugees cause crime?," World Development, Elsevier, vol. 154(C).
    18. Jiménez, Bruno & Rendon, Silvio, 2023. "Does employment protection unprotect workers? The labor market effects of job reinstatements in Peru," Labour Economics, Elsevier, vol. 80(C).
    19. Diogo G. C. Britto & Paolo Pinotti & Breno Sampaio, 2022. "The Effect of Job Loss and Unemployment Insurance on Crime in Brazil," Econometrica, Econometric Society, vol. 90(4), pages 1393-1423, July.
    20. Cl'ement de Chaisemartin & Xavier D'Haultf{oe}uille, 2021. "Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey," Papers 2112.04565, arXiv.org, revised Jun 2022.

    More about this item

    Keywords

    Frequentist; Hypothesis; Posterior; Testing;
    All these keywords.

    JEL classification:

    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:240:y:2024:i:1:s030440762400023x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.