How Much Should We Trust Modern Difference-in-Differences Estimates?

When do modern difference-in-differences (DID)-style methods work for empirical political science? Scholars exploit the staggered roll-out of policies like election regulation, civil service reform, and healthcare across places to estimate causal effects - often using the two-way fixed effects (TWFE) estimator. However, recent literature has highlighted the TWFE estimator's bias in the presence of heterogeneous treatment effects and tendency to make ``forbidden comparisons" between treated units. In response, scholars have increasingly turned to modern DID estimators that promise greater robustness to real-world data problems. This paper asks how well these modern methods work for for the empirical settings and sample sizes commonly used in political science, with the U.S. states as the running example. In particular, it provides a simulation study of the performance of seven DID methods under either constant or heterogeneous effects, in an N=50 setting that mimics the American federalism natural experiment. I find that many modern methods (1) produce confidence intervals that do not include the true average effect at the specified rate and (2) are underpowered. I show that many cases of coverage problems with modern DID estimators can be addressed using the block bootstrap to estimate standard errors. However, I also show that even where identification and estimation are straightforward, the fifty-state sample poses a power problem without large average effect sizes - at least 0.5 standard deviations. I illustrate the challenges of DID research with the fifty-state panel in the case of estimating the effects of strict voter identification laws on voter turnout.