IDEAS home Printed from https://ideas.repec.org/p/ces/econpb/_85.html

Evidence Under Scrutiny: How Reliable Is the Research Backing Public Policy?

Author

Listed:
  • Abel Brodeur
  • Nikolai Cook
  • Derek Mikola
  • Maddalena Totarelli
  • et. al.

Abstract

Key Messages85 percent of published economics and political science findings can be independently verifiedOne in four significant results disappear under different reasonable choices, and one in four studies contain undetected coding errorsPublished results cluster just above significance thresholds, which is evidence of publication biasPolicymakers should rely on independently reproduced evidence and avoid scaling up programs based on a single studyJournals and research entities should expand open data and robustness disclosure mandates

Suggested Citation

  • Abel Brodeur & Nikolai Cook & Derek Mikola & Maddalena Totarelli & et. al., 2026. "Evidence Under Scrutiny: How Reliable Is the Research Backing Public Policy?," EconPol Policy Brief 85, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    • Handle: RePEc:ces:econpb:_85
    as

    Download full text from publisher

    File URL: https://www.ifo.de/DocDL/EconPol_Policy_Brief_85_0.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Abel Brodeur & Nikolai Cook & Carina Neisser, 2024. "p-Hacking, Data type and Data-Sharing Policy," The Economic Journal, Royal Economic Society, vol. 134(659), pages 985-1018.
    2. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    3. Paul Gertler & Sebastian Galiani & Mauricio Romero, 2018. "How to make replication the norm," Nature, Nature, vol. 554(7693), pages 417-419, February.
    4. Abel Brodeur & Derek Mikola & Nikolai Cook & Lenka Fiala & Thomas Brailey & Ryan Briggs & Alexandra de Gendre & Yannick Dupraz & Jacopo Gabani & Romain Gauriot & Joanne Haddad & Goncalo Lima & Jörg An, 2026. "Reproducibility and robustness of economics and political science research," Nature, Nature, vol. 652(8108), pages 151-156, April.
    5. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    6. Abel Brodeur & Derek Mikola & Nikolai Cook & Lenka Fiala & Thomas Brailey & Ryan Briggs & Alexandra de Gendre & Yannick Dupraz & Jacopo Gabani & Romain Gauriot & Joanne Haddad & Goncalo Lima & Jörg An, 2026. "Reproducibility and robustness of economics and political science research," Nature, Nature, vol. 652(8108), pages 151-156, April.
    7. Andrew C. Chang & Phillip Li, 2022. "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Notâ€," Critical Finance Review, now publishers, vol. 11(1), pages 185-206, February.
    8. Lars Vilhuber, 2026. "Report of the AEA Data Editor," AEA Papers and Proceedings, American Economic Association, vol. 116, pages 890-904, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P & Hepplewhite, Matthew & de la Guardia, Fernando Hoces, 2024. "Promoting Reproducibility and Replicability in Political Science," Department of Economics, Working Paper Series qt23n3n3dg, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    2. Montpetit, Sébastien, 2026. "Behind the Veil of Origin: Revisiting the Impacts of the French Headscarf Ban in Schools," I4R Discussion Paper Series 293, The Institute for Replication (I4R).
    3. Brodeur, Abel & Mikola, Derek & Cook, Nikolai, 2024. "Mass Reproducibility and Replicability: A New Hope," IZA Discussion Papers 16912, IZA Network @ LISER.
    4. Montpetit, Sébastien, 2026. "Behind the Veil of Origin: Revisiting the Impacts of the French Headscarf Ban in Schools," The Warwick Economics Research Paper Series (TWERPS) 1610, University of Warwick, Department of Economics.
    5. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    6. Christophe Hurlin & Jean-Edouard Colliard & Christophe Pérignon, 2024. "The Economics of Computational Reproducibility," Working Papers hal-04759240, HAL.
    7. Stevenson, Megan T. & Fischman, Joshua B., 2026. "Humans in the Loop: The Next Frontier in the Credibility Revolution," I4R Discussion Paper Series 296, The Institute for Replication (I4R).
    8. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    9. Anthony Doucouliagos & Hristos Doucouliagos & T. D. Stanley, 2024. "Power and bias in industrial relations research," British Journal of Industrial Relations, London School of Economics, vol. 62(1), pages 3-27, March.
    10. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    11. Costanza Naguib, 2025. "Does single-blind review encourage or discourage p-hacking?," Diskussionsschriften dp2504, Universitaet Bern, Departement Volkswirtschaft.
    12. Bruns, Stephan B. & Doucouliagos, Anthony & Doucouliagos, Hristos & König, Johannes & Stanley, T.D. & Zigova, Katarina, 2026. "The delayed acceptance of female research in economics," European Economic Review, Elsevier, vol. 182(C).
    13. Jörg Ankel‐Peters & Nathan Fiala & Florian Neubauer, 2025. "Is economics self‐correcting? Replications in the American Economic Review," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 463-485, April.
    14. Christophe Pérignon & Olivier Akmansoy & Christophe Hurlin & Anna Dreber & Felix Holzmeister & Juergen Huber & Magnus Johanneson & Michael Kirchler & Albert J Menkveld & Michael Razen & Utz Weitzel, 2022. "Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance," Working Papers hal-03810013, HAL.
    15. repec:osf:osfxxx:mydzv_v1 is not listed on IDEAS
    16. Lukas Fink & Jan Marcus, 2025. "Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 357-386, April.
    17. Anna Dreber & Magnus Johannesson, 2025. "A framework for evaluating reproducibility and replicability in economics," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 338-356, April.
    18. repec:osf:osfxxx:u4vgs_v1 is not listed on IDEAS
    19. Fernando Hoces de la Guardia & Sean Grant & Edward Miguel, 2021. "A framework for open policy analysis," Science and Public Policy, Oxford University Press, vol. 48(2), pages 154-163.
    20. Garg, Prashant & Fetzer, Thiemo, 2024. "Causal Claims in Economics," OSF Preprints u4vgs, Center for Open Science.
    21. Edward Miguel, 2021. "Evidence on Research Transparency in Economics," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 193-214, Summer.
    22. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:econpb:_85. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/ifooode.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.