IDEAS home Printed from https://ideas.repec.org/a/now/jnlcfr/104.00000053.html
   My bibliography  Save this article

Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Notâ€

Author

Listed:
  • Andrew C. Chang
  • Phillip Li

Abstract

We attempted to replicate 67 macroeconomic papers published in 13 well-regarded economics journals using author-provided replication files that included both data and code by following a preanalysis plan. Aside from six papers that used confidential data, we obtained data and code replication files for 29 of 35 papers (83%) that were required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that were not required to provide data and code replication files. Defining replication success as our ability to use the author-provided data and code files to produce the key qualitative conclusions of the original paper, we successfully replicated 22 of 67 papers (33%) without contacting the authors. Excluding the six papers that used confidential data and the two papers that used software we did not possess, we replicated 29 of 59 papers (49%) with assistance from the authors. Because we were able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is often not replicable. We conclude with recommendations on improving replication of economics research.

Suggested Citation

  • Andrew C. Chang & Phillip Li, 2022. "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Notâ€," Critical Finance Review, now publishers, vol. 11(1), pages 185-206, February.
    • Handle: RePEc:now:jnlcfr:104.00000053
    DOI: 10.1561/104.00000053
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1561/104.00000053
    Download Restriction: no
    File URL: https://libkey.io/10.1561/104.00000053?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John T. Dalton & Lillian R. Gaeto, 2022. "Schumpeter vs. Keynes redux: “Still not dead”," Southern Economic Journal, John Wiley & Sons, vol. 89(2), pages 569-592, October.
    2. Andrew C. Chang & Trace J. Levinson, 2023. "Raiders of the lost high‐frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(1), pages 88-104, January.
    3. Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S. & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P. & Hepplewhite, Matthew & Hoces de la Guardia, Fernan, 2024. "Promoting Reproducibility and Replicability in Political Science," I4R Discussion Paper Series 100, The Institute for Replication (I4R).

    More about this item

    Keywords

    Data and code files; Gross domestic product; GDP; Macroeconomics; National income and product accounts; Preanalysis; Publication bias; Replication;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General
    • C82 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Macroeconomic Data; Data Access
    • C87 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Econometric Software
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • E01 - Macroeconomics and Monetary Economics - - General - - - Measurement and Data on National Income and Product Accounts and Wealth; Environmental Accounts

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:now:jnlcfr:104.00000053. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Lucy Wiseman (email available below). General contact details of provider: http://www.nowpublishers.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.