IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2309.15983.html

Causal Panel Analysis under Parallel Trends: Lessons from a Large Reanalysis Study

Author

Listed:
  • Albert Chiu
  • Xingchen Lan
  • Ziyi Liu
  • Yiqing Xu

Abstract

Two-way fixed effects (TWFE) models are widely used in political science to establish causality, but recent methodological discussions highlight their limitations under heterogeneous treatment effects (HTE) and violations of the parallel trends (PT) assumption. This growing literature has introduced numerous new estimators and procedures, causing confusion among researchers about the reliability of existing results and best practices. To address these concerns, we replicated and reanalyzed 49 studies from leading journals using TWFE models for observational panel data with binary treatments. Using six HTE-robust estimators, diagnostic tests, and sensitivity analyses, we find: (i) HTE-robust estimators yield qualitatively similar but highly variable results; (ii) while a few studies show clear signs of PT violations, many lack evidence to support this assumption; and (iii) many studies are underpowered when accounting for HTE and potential PT violations. We emphasize the importance of strong research designs and rigorous validation of key identifying assumptions.

Suggested Citation

  • Albert Chiu & Xingchen Lan & Ziyi Liu & Yiqing Xu, 2023. "Causal Panel Analysis under Parallel Trends: Lessons from a Large Reanalysis Study," Papers 2309.15983, arXiv.org, revised Jun 2025.
    • Handle: RePEc:arx:papers:2309.15983
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2309.15983
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Junyan Jiang, 2018. "Making Bureaucracy Work: Patronage Networks, Performance Incentives, and Economic Development in China," American Journal of Political Science, John Wiley & Sons, vol. 62(4), pages 982-999, October.
    2. Martin Bisgaard & Rune Slothuus, 2018. "Partisan Elites as Culprits? How Party Cues Shape Partisan Perceptual Gaps," American Journal of Political Science, John Wiley & Sons, vol. 62(2), pages 456-469, April.
    3. Alexander Fouirnaies & Andrew B. Hall, 2018. "How Do Interest Groups Seek Access to Committees?," American Journal of Political Science, John Wiley & Sons, vol. 62(1), pages 132-147, January.
    4. Kirill Borusyak & Xavier Jaravel & Jann Spiess, 2024. "Revisiting Event-Study Designs: Robust and Efficient Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(6), pages 3253-3285.
    5. Callaway, Brantly & Karami, Sonia, 2023. "Treatment effects in interactive fixed effects models with a small number of time periods," Journal of Econometrics, Elsevier, vol. 233(1), pages 184-208.
    6. Greg Distelhorst & Richard M. Locke, 2018. "Does Compliance Pay? Social Standards and Firm‐Level Trade," American Journal of Political Science, John Wiley & Sons, vol. 62(3), pages 695-711, July.
    7. Andrew J. Clarke, 2020. "Party Sub‐Brands and American Party Factions," American Journal of Political Science, John Wiley & Sons, vol. 64(3), pages 452-470, July.
    8. Beck, Nathaniel & Katz, Jonathan N., 2001. "Throwing Out the Baby with the Bath Water: A Comment on Green, Kim, and Yoon," International Organization, Cambridge University Press, vol. 55(2), pages 487-495, April.
    9. Dmitry Arkhangelsky & Guido W. Imbens, 2019. "Doubly Robust Identification for Causal Panel Data Models," Papers 1909.09412, arXiv.org, revised Feb 2022.
    10. Jens Hainmueller & Dominik Hangartner, 2019. "Does Direct Democracy Hurt Immigrant Minorities? Evidence from Naturalization Decisions in Switzerland," American Journal of Political Science, John Wiley & Sons, vol. 63(3), pages 530-547, July.
    11. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    12. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    13. Hainmueller, Jens & Mummolo, Jonathan & Xu, Yiqing, 2019. "How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice," Political Analysis, Cambridge University Press, vol. 27(2), pages 163-192, April.
    14. Jushan Bai & Serena Ng, 2021. "Matrix Completion, Counterfactuals, and Factor Analysis of Missing Data," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1746-1763, October.
    15. Payson, Julia A., 2020. "The Partisan Logic of City Mobilization: Evidence from State Lobbying Disclosures," American Political Science Review, Cambridge University Press, vol. 114(3), pages 677-690, August.
    16. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    17. Dmitry Arkhangelsky & Guido W Imbens, 2022. "Doubly robust identification for causal panel data models [Sufficient statistics for unobserved heterogeneity in structural dynamic logit models]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 649-674.
    18. Andrew C. Eggers & Anthony Fowler & Jens Hainmueller & Andrew B. Hall & James M. Snyder, 2015. "On the Validity of the Regression Discontinuity Design for Estimating Electoral Effects: New Evidence from Over 40,000 Close Races," American Journal of Political Science, John Wiley & Sons, vol. 59(1), pages 259-274, January.
    19. Baker, Andrew C. & Larcker, David F. & Wang, Charles C.Y., 2022. "How much should we trust staggered difference-in-differences estimates?," Journal of Financial Economics, Elsevier, vol. 144(2), pages 370-395.
    20. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    21. Alexander Fouirnaies, 2018. "When Are Agenda Setters Valuable?," American Journal of Political Science, John Wiley & Sons, vol. 62(1), pages 176-191, January.
    22. Trounstine, Jessica, 2020. "The Geography of Inequality: How Land Use Regulation Produces Segregation," American Political Science Review, Cambridge University Press, vol. 114(2), pages 443-455, May.
    23. Daniel Bischof & Markus Wagner, 2019. "Do Voters Polarize When Radical Parties Enter Parliament?," American Journal of Political Science, John Wiley & Sons, vol. 63(4), pages 888-904, October.
    24. Fouirnaies, Alexander & Hall, Andrew B., 2022. "How Do Electoral Incentives Affect Legislator Behavior? Evidence from U.S. State Legislatures," American Political Science Review, Cambridge University Press, vol. 116(2), pages 662-676, May.
    25. Jonathan Roth, 2022. "Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends," American Economic Review: Insights, American Economic Association, vol. 4(3), pages 305-322, September.
    26. Paglayan, Agustina S., 2022. "Education or Indoctrination? The Violent Origins of Public School Systems in an Era of State-Building," American Political Science Review, Cambridge University Press, vol. 116(4), pages 1242-1257, November.
    27. Kosuke Imai & In Song Kim, 2019. "When Should We Use Unit Fixed Effects Regression Models for Causal Inference with Longitudinal Data?," American Journal of Political Science, John Wiley & Sons, vol. 63(2), pages 467-490, April.
    28. Magaloni, Beatriz & Franco-Vivanco, Edgar & Melo, Vanessa, 2020. "Killing in the Slums: Social Order, Criminal Governance, and Police Violence in Rio de Janeiro," American Political Science Review, Cambridge University Press, vol. 114(2), pages 552-572, May.
    29. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    30. Plümper, Thomas & Troeger, Vera E., 2019. "Not so Harmless After All: The Fixed-Effects Model," Political Analysis, Cambridge University Press, vol. 27(1), pages 21-45, January.
    31. Lall, Ranjit, 2016. "How Multiple Imputation Makes a Difference," Political Analysis, Cambridge University Press, vol. 24(4), pages 414-433.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bogatyrev, Konstantin & Stoetzer, Lukas, 2024. "Synthetic Control Methods for Proportions," OSF Preprints brhd3, Center for Open Science.
    2. Ulbing, Philipp, 2024. "The Zero Lower Bound on Household Deposit Rates: Not As Binding As We Thought," VfS Annual Conference 2024 (Berlin): Upcoming Labor Market Challenges 302353, Verein für Socialpolitik / German Economic Association.
    3. Rüttenauer, Tobias & Kapelle, Nicole, 2024. "Panel Data Analysis," SocArXiv 3mfzq, Center for Open Science.
    4. Bach, Philipp & Klaaßen, Sven & Kueck, Jannis & Mattes, Mara & Spindler, Martin, 2025. "Sensitivity analysis for treatment effects in difference-in-differences models using Riesz Rrepresentation," Discussion Papers 2025/7, Free University Berlin, School of Business & Economics.
    5. Philipp Bach & Sven Klaassen & Jannis Kueck & Mara Mattes & Martin Spindler, 2025. "Sensitivity Analysis for Treatment Effects in Difference-in-Differences Models using Riesz Representation," Papers 2510.09064, arXiv.org.
    6. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    7. Strezhnev, Anton, 2024. "Group-specific linear trends and the triple-differences in time design," SocArXiv dg5ps, Center for Open Science.
    8. Weiss, Amanda, 2024. "How Much Should We Trust Modern Difference-in-Differences Estimates?," OSF Preprints bqmws, Center for Open Science.
    9. Strebel, Michael A., 2025. "Are municipal mergers an antidote to recruitment problems for local political office?," SocArXiv cyrtm_v1, Center for Open Science.
    10. Tobias Ruttenauer & Ozan Aksoy, 2024. "When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators," Papers 2402.09928, arXiv.org, revised Jun 2025.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    2. Gregory Faletto, 2023. "Fused Extended Two-Way Fixed Effects for Difference-in-Differences With Staggered Adoptions," Papers 2312.05985, arXiv.org, revised Apr 2025.
    3. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    4. Chen, Xing & Huang, Zhijian (James) & Li, Zhuo & Wen, Fenghua, 2025. "Impact from a distance: Emotionally attached-place disasters and corporate risk-taking," Journal of Corporate Finance, Elsevier, vol. 93(C).
    5. repec:osf:osfxxx:ac5ru_v1 is not listed on IDEAS
    6. Benjamin W. Arold & Ludger Woessmann & Larissa Zierow, 2022. "Can Schools Change Religious Attitudes? Evidence from German State Reforms of Compulsory Religious Education," CESifo Working Paper Series 9504, CESifo.
    7. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    8. Ben Deaner & Chen-Wei Hsiang & Andrei Zeleneev, 2025. "Inferring Treatment Effects in Large Panels by Uncovering Latent Similarities," Papers 2503.20769, arXiv.org, revised Mar 2025.
    9. Brehm, Johannes & Pestel, Nico & Schaffner, Sandra & Schmitz, Laura, 2025. "From Low Emission Zone to academic track: Environmental policy effects on educational achievement in elementary school," Journal of Environmental Economics and Management, Elsevier, vol. 132(C).
    10. Coraggio, Luca & Pagano, Marco & Scognamiglio, Annalisa & Tåg, Joacim, 2025. "JAQ of all trades: Job mismatch, firm productivity and managerial quality," Journal of Financial Economics, Elsevier, vol. 164(C).
    11. Mark Kattenberg & Bas Scheer & Jurre Thiel, 2023. "Causal forests with fixed effects for treatment effect heterogeneity in difference-in-differences," CPB Discussion Paper 452, CPB Netherlands Bureau for Economic Policy Analysis.
    12. Li, Pei & Liu, Kaihao & Lu, Yi & Peng, Lu, 2025. "Organizing regulatory structure and local air quality: Evidence from the environmental vertical management reform in China," Journal of Comparative Economics, Elsevier, vol. 53(1), pages 139-164.
    13. Chen, Jidong & Shi, Xinzheng & Zhang, Ming-ang & Zhang, Sihan, 2024. "Centralization of environmental administration and air pollution: Evidence from China," Journal of Environmental Economics and Management, Elsevier, vol. 126(C).
    14. Zhu, Junjie & Guo, Hongfeng, 2025. "Does the development of high-speed rail benefit carbon emissions reduction?," Transport Policy, Elsevier, vol. 172(C).
    15. Matthias Breuer & Ed Dehaan, 2024. "Using and Interpreting Fixed Effects Models," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 62(4), pages 1183-1226, September.
    16. Fayssal Ayad & Ahlam Delmadji, 2025. "Fifty shades of sustainable development: a global study," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 27(7), pages 16739-16771, July.
    17. repec:ags:aaea22:335779 is not listed on IDEAS
    18. Millar, Melanie I. & White, Roger M., 2024. "Do residential property assessed clean energy (PACE) financing programs affect local house price growth?," Journal of Environmental Economics and Management, Elsevier, vol. 124(C).
    19. Stefan Szymanski, 2023. "Anticipating the honeymoon: Event study estimation of new stadium effects in Major League Baseball using the imputation method," Economic Inquiry, Western Economic Association International, vol. 61(4), pages 1077-1102, October.
    20. Li, Tongxia & Lu, Chun & Xu, Lei, 2025. "Access to finance and cost stickiness: Evidence from anti-recharacterization laws," Advances in accounting, Elsevier, vol. 68(C).
    21. Traviss Cassidy & Mark Dincecco & Ugo Antonio Troiano, 2024. "The Introduction of the Income Tax, Fiscal Capacity, and Migration: Evidence from US States," American Economic Journal: Economic Policy, American Economic Association, vol. 16(1), pages 359-393, February.
    22. Hollenbach, Florian M & Egerod, Benjamin, 2024. "How many is enough? Sample Size in Staggered Difference-in-Differences Designs," OSF Preprints ac5ru, Center for Open Science.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2309.15983. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.