IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/h29xn_v1.html
   My bibliography  Save this paper

Estimating the extent of inflated significance in economics

Author

Listed:
  • Bruns, Stephan
  • Deressa, Teshome Kebede
  • Stanley, T D
  • Doucouliagos, Chris
  • Ioannidis, John P.A.

Abstract

Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as being non-significant. We estimate the extent of inflated significance to range between 56.2% and 71.3% of the significant p-values. Subsample analysis suggests that the extent of inflated significance is reduced in research fields that use experimental designs, analyze microeconomics research questions and have at least some adequately powered studies.

Suggested Citation

  • Bruns, Stephan & Deressa, Teshome Kebede & Stanley, T D & Doucouliagos, Chris & Ioannidis, John P.A., 2022. "Estimating the extent of inflated significance in economics," MetaArXiv h29xn_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:h29xn_v1
    DOI: 10.31219/osf.io/h29xn_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/622b744153a4e80277516456/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/h29xn_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Burlig, Fiona & Preonas, Louis & Woerman, Matt, 2020. "Panel data and experimental design," Journal of Development Economics, Elsevier, vol. 144(C).
    2. Alexis H. Villacis & Jeffrey R. Alwang & Victor Barrera & Juan Dominguez, 2022. "Prices, specialty varieties, and postharvest practices: Insights from cacao value chains in Ecuador," Agribusiness, John Wiley & Sons, Ltd., vol. 38(2), pages 426-458, April.
    3. Ashesh Rambachan & Neil Shephard, 2019. "When do common time series estimands have nonparametric causal meaning?," Papers 1903.01637, arXiv.org, revised Jan 2025.
    4. Sergey Alexeev, 2023. "Technical change and wage premiums amongst skilled labour: Evidence from the economic transition," Economics of Transition and Institutional Change, John Wiley & Sons, vol. 31(1), pages 189-216, January.
    5. Millimet, Daniel L. & Bellemare, Marc, 2023. "Fixed Effects and Causal Inference," IZA Discussion Papers 16202, Institute of Labor Economics (IZA).
    6. Nepal, Rabindra & Best, Rohan & Taylor, Madeline, 2023. "Strategies for reducing ethnic inequality in energy outcomes: A Nepalese example," Energy Economics, Elsevier, vol. 126(C).
    7. Mayoral, Laura & Ray, Debraj, 2022. "Groups in conflict: Private and public prizes," Journal of Development Economics, Elsevier, vol. 154(C).
    8. Aguilar-Gomez, Sandra & Benshaul-Tolonen, Anja, 2023. "The evolution and persistence of women's roles: Evidence from the Gold Rush," Journal of Economic Behavior & Organization, Elsevier, vol. 215(C), pages 364-381.
    9. Christoph Engel, 2016. "Experimental Criminal Law. A Survey of Contributions from Law, Economics and Criminology," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2016_07, Max Planck Institute for Research on Collective Goods.
    10. Czajkowski, Mikołaj & Zagórska, Katarzyna & Letki, Natalia & Tryjanowski, Piotr & Wąs, Adam, 2021. "Drivers of farmers’ willingness to adopt extensive farming practices in a globally important bird area," Land Use Policy, Elsevier, vol. 107(C).
    11. Ryan Abman & Clark Lundberg & Michele Ruta, 2024. "The Effectiveness of Environmental Provisions in Regional Trade Agreements," Journal of the European Economic Association, European Economic Association, vol. 22(6), pages 2507-2548.
    12. Mikel Bedayo & Raquel Vegas & Gabriel Jiménez & José-Luis Peydró, 2020. "Screening and Loan Origination Time: Lending Standards, Loan Defaults and Bank Failures," Working Papers 1215, Barcelona School of Economics.
    13. Michael A. Clemens, 2016. "Losing our minds? New research directions on skilled emigration and development," International Journal of Manpower, Emerald Group Publishing Limited, vol. 37(7), pages 1227-1248, October.
    14. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    15. Sónia Cabral & Joana Garcia & Raquel Miranda & Susana Peralta & João Pereira dos Santos, 2024. "Treasure Islands, Real Jobs? Workers and Anti-Avoidance Policies in a Tax Paradise," Working Papers w202416, Banco de Portugal, Economics and Research Department.
    16. Francesco Fallucchi & Jan Niederreiter & Massimo Riccaboni, 2021. "Learning and dropout in contests: an experimental approach," Theory and Decision, Springer, vol. 90(2), pages 245-278, March.
    17. Allais, Olivier & Leroy, Pascal & Mink, Julia, 2020. "Changes in food purchases at retirement in France," Food Policy, Elsevier, vol. 90(C).
    18. Christopher Conlon & Julie Holland Mortimer, 2021. "Empirical properties of diversion ratios," RAND Journal of Economics, RAND Corporation, vol. 52(4), pages 693-726, December.
    19. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    20. Tal Gross & Timothy J. Layton & Daniel Prinz, 2022. "The Liquidity Sensitivity of Healthcare Consumption: Evidence from Social Security Payments," American Economic Review: Insights, American Economic Association, vol. 4(2), pages 175-190, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:h29xn_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.