IDEAS home Printed from https://ideas.repec.org/p/fip/fedgfe/103446.html

An Evaluation of Difference-in-Differences Methods Using Placebo Event Studies

Author

Listed:

Abstract

Researchers are faced with the choice of which of the many recently developed difference-in-differences methods to use in practice. To assess these estimators' relative performance for single-unit event studies, we conduct 134,000+ state-level placebo event studies across 13 estimators. We find that no single method dominates. Performance is context-dependent, with synthetic-control-like methods sometimes outperforming and sometimes underperforming two-way-fixed-effect-like and matching methods. Performance also varies across states at least as much as it does across estimators. Our results highlight the need for practitioners to conduct placebo tests to understand the performance of methods in their research context.

Suggested Citation

  • John Coglianese & Jade A. Fang, 2026. "An Evaluation of Difference-in-Differences Methods Using Placebo Event Studies," Finance and Economics Discussion Series 2026-045, Board of Governors of the Federal Reserve System (U.S.).
    • Handle: RePEc:fip:fedgfe:103446
    DOI: 10.17016/FEDS.2026.045
    as

    Download full text from publisher

    File URL: https://www.federalreserve.gov/econres/feds/files/2026045pap.pdf
    File Function: Full text
    Download Restriction: no
    File URL: https://libkey.io/10.17016/FEDS.2026.045?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Andrew Baker & Brantly Callaway & Scott Cunningham & Andrew Goodman-Bacon & Pedro H. C. Sant'Anna, 2026. "Difference-in-Differences Designs: A Practitioner's Guide," Journal of Economic Literature, American Economic Association, vol. 64(2), pages 498-557, June.
    2. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    3. Xu, Yiqing, 2017. "Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models," Political Analysis, Cambridge University Press, vol. 25(1), pages 57-76, January.
    4. Kirill Borusyak & Xavier Jaravel & Jann Spiess, 2024. "Revisiting Event-Study Designs: Robust and Efficient Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(6), pages 3253-3285.
    5. William Wascher & David Neumark, 2000. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania: Comment," American Economic Review, American Economic Association, vol. 90(5), pages 1362-1396, December.
    6. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    7. Richard Hornbeck & Enrico Moretti, 2024. "Estimating Who Benefits from Productivity Growth: Local and Distant Effects of City Productivity Growth on Wages, Rents, and Inequality," The Review of Economics and Statistics, MIT Press, vol. 106(3), pages 587-607, May.
    8. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    9. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    10. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    11. Arindrajit Dube & Daniele Girardi & Òscar Jordà & Alan M. Taylor, 2025. "A Local Projections Approach to Difference‐in‐Differences," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(7), pages 741-758, November.
    12. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    13. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    14. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    15. Eli Ben‐Michael & Avi Feller & Jesse Rothstein, 2022. "Synthetic controls with staggered adoption," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 351-381, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    2. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2026. "Estimating Causal Effects With Observational Data: Guidelines for Agricultural and Applied Economists," Journal of Agricultural Economics, Wiley Blackwell, vol. 77(2), pages 356-382, June.
    3. Anish Agarwal & Sukjin Han & Dwaipayan Saha & Vasilis Syrgkanis & Haeyeon Yoon, 2026. "Synthetic Blips: Generalizing Synthetic Controls for Dynamic Treatment Effects," Bristol Economics Discussion Papers 26/834, School of Economics, University of Bristol, UK.
    4. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    5. Nyborg, Kjell G. & Woschitz, Jiri, 2025. "Robust difference-in-differences analysis when there is a term structure," Journal of Financial Economics, Elsevier, vol. 170(C).
    6. Takahiro Hoshino & Makoto Nakakita, 2026. "Shared-Donor Inference for Fixed-Set Heterogeneity in Synthetic Difference-in-Differences," Papers 2607.08324, arXiv.org, revised Sep 2026.
    7. Dalia Ghanem & Pedro H. C. Sant'Anna & Kaspar Wüthrich, 2022. "Selection and Parallel Trends," CESifo Working Paper Series 9910, CESifo.
    8. Yixiao Sun & Haitian Xie & Yuhang Zhang, 2025. "Difference-in-Differences Meets Synthetic Control: Doubly Robust Identification and Estimation," Papers 2503.11375, arXiv.org, revised Sep 2025.
    9. van Rijn, Jordan, 2024. "The effects of membership expansion on credit union risk and returns," International Review of Financial Analysis, Elsevier, vol. 93(C).
    10. Jack (Peiyao) Ma & Andrea Mantovani & Carlo Reggiani & Annette Broocks & Néstor Duch-Brown, 2024. "The Price Effects of Prohibiting Price Parity Clauses: Evidence from International Hotel Groups," Economics Series Working Papers 1043, University of Oxford, Department of Economics.
    11. Hans-Bernd Schaefer & Rok Spruk, 2024. "Islamic Law, Western European Law and the Roots of Middle East's Long Divergence: a Comparative Empirical Investigation (800-1600)," Papers 2401.14435, arXiv.org, revised Mar 2024.
    12. Cl'ement de Chaisemartin & Xavier D'Haultf{oe}uille, 2025. "Treatment-Effect Estimation in Complex Designs under a Parallel-trends Assumption," Papers 2508.07808, arXiv.org, revised Dec 2025.
    13. Ratzanyel Rinc'on & Kyungchul Song, 2025. "Causal Inference with Groupwise Matching," Papers 2510.26106, arXiv.org, revised Mar 2026.
    14. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    15. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.
    16. Luya Wang & Jeffrey S. Racine & Qiaoyu Wang, 2025. "Bootstrap inference on a factor model based average treatment effects estimator," Econometric Reviews, Taylor & Francis Journals, vol. 44(1), pages 80-89, January.
    17. Yu, Yantuan & Zhang, Ning, 2025. "Towards low-carbon development through integration of technology and finance: Quasi-experimental evidence from China," Research in International Business and Finance, Elsevier, vol. 80(C).
    18. Justin C. Wiltshire, 2023. "Walmart Supercenters and Monopsony Power: How A Large, Low-Wage Employer Impacts Local Labor Markets," Department Discussion Papers 2304, Department of Economics, University of Victoria.
    19. Elena Kotyrlo & Hryhorii M. Kalachyhin, 2023. "The effects of India's bilateral investment treaties termination on foreign direct investment inflows," Economics of Transition and Institutional Change, John Wiley & Sons, vol. 31(4), pages 1007-1033, October.
    20. Kyunghoon Ban & D'esir'e K'edagni, 2022. "Robust Difference-in-differences Models," Papers 2211.06710, arXiv.org, revised Aug 2023.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedgfe:103446. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ryan Wolfslayer ; Keisha Fournillier (email available below). General contact details of provider: https://edirc.repec.org/data/frbgvus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.