IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/315.html

How did empirical economic research improve over the last decades: Recent trends and prospects

Author

Listed:
  • Brodeur, Abel
  • Bruns, Stephan B.
  • Herwartz, Helmut
  • Islam, Chris-Gabriel

Abstract

We map the evolution of economics research from 1998 to 2019 to assess progress and identify remaining challenges. Using text mining of 30,675 full-text articles and an in-depth analysis of 578,132 statistical tests from 3,746 articles across a broad range of journals, we document changes in research practices over time. We highlight the influence of the credibility revolution and the open science movement. Awareness of reproducibility has increased modestly, largely driven by journal adoption of data and code sharing policies. At the same time, reliance on non-public data has grown, especially among authors at top-five universities, posing new challenges for transparency. Sample sizes have increased substantially, while reported effect sizes have declined, consistent with reduced exaggeration and potentially more policy-relevant estimates. Econometric methods and causal inference practices have become more sophisticated, and articles now report more tables and statistical tests. However, pre-analysis plans, power analyses accompanying null results, and corrections for multiple hypothesis testing remain uncommon. Article length has remained stable or declined slightly, and authors from top-five universities continue to represent a large and stable share of publications in top journals. Our findings provide a baseline for tracking future developments and for guiding efforts to further strengthen the credibility and transparency of economics research.

Suggested Citation

  • Brodeur, Abel & Bruns, Stephan B. & Herwartz, Helmut & Islam, Chris-Gabriel, 2026. "How did empirical economic research improve over the last decades: Recent trends and prospects," I4R Discussion Paper Series 315, The Institute for Replication (I4R).
  • Handle: RePEc:zbw:i4rdps:315
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/342985/1/I4R-DP315.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ziliak, Stephen T. & McCloskey, Deirdre N., 2004. "Size matters: the standard error of regressions in the American Economic Review," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 527-546, November.
    2. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Marcus R. Munafò & George Davey Smith, 2018. "Robust research needs many lines of evidence," Nature, Nature, vol. 553(7689), pages 399-401, January.
    5. David Card & Stefano DellaVigna & Patricia Funk & Nagore Iriberri, 2020. "Are Referees and Editors in Economics Gender Neutral?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 135(1), pages 269-327.
    6. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    7. Stephan B. Bruns & Christian Gross & David I. Stern, 2014. "Is There Really Granger Causality between Energy Use and Output?," The Energy Journal, , vol. 35(4), pages 101-134, October.
    8. Ana Rute Cardoso & Paulo Guimarães & Klaus F. Zimmermann, 2010. "Trends in Economic Research: An International Perspective," Kyklos, Wiley Blackwell, vol. 63(4), pages 479-494, November.
    9. Peter Pütz & Stephan B. Bruns, 2021. "The (Non‐)Significance Of Reporting Errors In Economics: Evidence From Three Top Journals," Journal of Economic Surveys, Wiley Blackwell, vol. 35(1), pages 348-373, February.
    10. Joel Ferguson & Rebecca Littman & Garret Christensen & Elizabeth Levy Paluck & Nicholas Swanson & Zenan Wang & Edward Miguel & David Birke & John-Henry Pezzuto, 2023. "Survey of open science practices and attitudes in the social sciences," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    11. Liran Einav & Jonathan Levin, 2014. "The Data Revolution and Economic Analysis," Innovation Policy and the Economy, University of Chicago Press, vol. 14(1), pages 1-24.
    12. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    13. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    14. Abhijit Banerjee & Esther Duflo & Amy Finkelstein & Lawrence F. Katz & Benjamin A. Olken & Anja Sautmann, 2020. "In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics," NBER Working Papers 26993, National Bureau of Economic Research, Inc.
    15. Scott, Loren C & Mitias, Peter M, 1996. "Trends in Rankings of Economics Departments in the U.S.: An Update," Economic Inquiry, Western Economic Association International, vol. 34(2), pages 378-400, April.
    16. repec:bla:jecsur:v:16:y:2002:i:4:p:569-89 is not listed on IDEAS
    17. Guido W. Imbens, 2021. "Statistical Significance, p-Values, and the Reporting of Uncertainty," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 157-174, Summer.
    18. Paul Goldsmith-Pinkham, 2024. "Tracking the Credibility Revolution across Fields," Papers 2405.20604, arXiv.org, revised Jun 2024.
    19. Taisuke Imai & Tom A Rutter & Colin F Camerer, 2021. "Meta-Analysis of Present-Bias Estimation using Convex Time Budgets," The Economic Journal, Royal Economic Society, vol. 131(636), pages 1788-1814.
    20. Graham Elliott & Nikolay Kudrin & Kaspar Wüthrich, 2022. "Detecting p‐Hacking," Econometrica, Econometric Society, vol. 90(2), pages 887-906, March.
    21. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    22. Abel Brodeur & Derek Mikola & Nikolai Cook & Lenka Fiala & Thomas Brailey & Ryan Briggs & Alexandra de Gendre & Yannick Dupraz & Jacopo Gabani & Romain Gauriot & Joanne Haddad & Goncalo Lima & Jörg An, 2026. "Reproducibility and robustness of economics and political science research," Nature, Nature, vol. 652(8108), pages 151-156, April.
    23. Anne E Winkler & Sharon G Levin & Paula E Stephan & Wolfgang Gl&aauml;nzel, 2014. "Publishing Trends in Economics across Colleges and Universities, 1991–2007," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 40(4), pages 560-582, September.
    24. Hamermesh, Daniel S., 2025. "The Demographic and Research Styles of Economics Writing," IZA Discussion Papers 17863, IZA Network @ LISER.
    25. Edward Miguel, 2021. "Evidence on Research Transparency in Economics," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 193-214, Summer.
    26. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    27. repec:ejw:journl:v:1:y:2004:i:2:p:331-358 is not listed on IDEAS
    28. Abel Brodeur & Derek Mikola & Nikolai Cook & Lenka Fiala & Thomas Brailey & Ryan Briggs & Alexandra de Gendre & Yannick Dupraz & Jacopo Gabani & Romain Gauriot & Joanne Haddad & Goncalo Lima & Jörg An, 2026. "Reproducibility and robustness of economics and political science research," Nature, Nature, vol. 652(8108), pages 151-156, April.
    29. Nicholas G. Rupp, 2002. "The Publication Patterns of the Elite Economics Departments: 1995-2000," Eastern Economic Journal, Eastern Economic Association, vol. 28(4), pages 523-538, Fall.
    30. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    31. Daniel S. Hamermesh, 2013. "Six Decades of Top Economics Publishing: Who and How?," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 162-172, March.
    32. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    33. Eva Vivalt, 2019. "Specification Searching and Significance Inflation Across Time, Methods and Disciplines," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 81(4), pages 797-816, August.
    34. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    35. Bruns, Stephan B. & Doucouliagos, Anthony & Doucouliagos, Hristos & König, Johannes & Stanley, T.D. & Zigova, Katarina, 2026. "The delayed acceptance of female research in economics," European Economic Review, Elsevier, vol. 182(C).
    36. repec:osf:metaar:mbx62_v1 is not listed on IDEAS
    37. Bruns, Stephan & Herwartz, Helmut & Ioannidis, John P.A. & Islam, Chris-Gabriel & Raters, Fabian H. C., 2023. "Statistical reporting errors in economics," MetaArXiv mbx62, Center for Open Science.
    38. Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2024. "The Null Result Penalty," The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
    39. Vlaeminck, Sven, 2021. "Dawning of a New Age? Economics Journals’ Data Policies on the Test Bench," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 31(1), pages 1-29.
    40. Olivia Miske & Anna Lou Abatayo & Mason Daley & Mirka Dirzo & Nicholas Fox & Noah Haber & Krystal M. Hahn & Melissa Kline Struhl & Brinna Mawhinney & Priya Silverstein & Theresa Stankov & Andrew H. Ty, 2026. "Investigating the reproducibility of the social and behavioural sciences," Nature, Nature, vol. 652(8108), pages 126-134, April.
    41. repec:aen:journl:ej35-4-05 is not listed on IDEAS
    42. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    43. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    44. Andrew H. Tyner & Anna Lou Abatayo & Mason Daley & Samuel Field & Nicholas Fox & Noah A. Haber & Krystal M. Hahn & Melissa Kline Struhl & Brinna Mawhinney & Olivia Miske & Priya Silverstein & Courtney, 2026. "Investigating the replicability of the social and behavioural sciences," Nature, Nature, vol. 652(8108), pages 143-150, April.
    45. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    46. Andrew H. Briggs, 1999. "A Bayesian approach to stochastic cost‐effectiveness analysis," Health Economics, John Wiley & Sons, Ltd., vol. 8(3), pages 257-261, May.
    47. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    48. Peter E. Kennedy, 2002. "Sinning in the Basement: What are the Rules? The Ten Commandments of Applied Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 16(4), pages 569-589, September.
    49. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 533-575.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abel Brodeur & Nikolai Cook & Carina Neisser, 2024. "p-Hacking, Data type and Data-Sharing Policy," The Economic Journal, Royal Economic Society, vol. 134(659), pages 985-1018.
    2. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    3. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    4. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    5. Bruns, Stephan & Herwartz, Helmut & Ioannidis, John P.A. & Islam, Chris-Gabriel & Raters, Fabian H. C., 2023. "Statistical reporting errors in economics," MetaArXiv mbx62, Center for Open Science.
    6. Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "The Power of Tests for Detecting $p$-Hacking," Papers 2205.07950, arXiv.org, revised Aug 2025.
    7. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    8. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    9. Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T. D. Stanley & Pedro R. D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2025. "Conventional wisdom, meta‐analysis, and research revision in economics," Journal of Economic Surveys, Wiley Blackwell, vol. 39(3), pages 980-999, July.
    10. Brodeur, Abel & Cook, Nikolai & Hartley, Jonathan & Heyes, Anthony, 2022. "Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?," MetaArXiv uxf39, Center for Open Science.
    11. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    12. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    13. Bruns, Stephan B. & Doucouliagos, Anthony & Doucouliagos, Hristos & König, Johannes & Stanley, T.D. & Zigova, Katarina, 2026. "The delayed acceptance of female research in economics," European Economic Review, Elsevier, vol. 182(C).
    14. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    15. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    16. Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2024. "The Null Result Penalty," The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
    17. Brodeur, Abel & Cook, Nikolai M. & Heyes, Anthony & Wright, Taylor, 2025. "Media Stars: Statistical Significance and Research Impact," I4R Discussion Paper Series 254, The Institute for Replication (I4R).
    18. Graham Elliott & Nikolay Kudrin & Kaspar Wüthrich, 2022. "Detecting p‐Hacking," Econometrica, Econometric Society, vol. 90(2), pages 887-906, March.
    19. Anna Dreber & Magnus Johannesson & Yifan Yang, 2024. "Selective reporting of placebo tests in top economics journals," Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
    20. repec:osf:osfxxx:u4vgs_v1 is not listed on IDEAS
    21. Garg, Prashant & Fetzer, Thiemo, 2024. "Causal Claims in Economics," OSF Preprints u4vgs, Center for Open Science.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • A19 - General Economics and Teaching - - General Economics - - - Other
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:315. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.