How many is enough? Sample Size in Staggered Difference-in-Differences Designs

In difference-in-differences designs with staggered treatment timing and dynamic treatment effects, the two-way fixed effects estimator fails to recover an interpretable causal estimate. A large number of estimators have been proposed to remedy this issue. The flexibility of these estimators, however, increases their variance. This can lead to statistical tests with low statistical power. As a consequence, small effects are unlikely to be discovered. Additionally, under low power, if a statistically significant estimate is recovered, the estimate is often wrongly signed and/or greatly exaggerated. Using simulations on real-world data on US States, we show that effect sizes of 10 to 15% are necessary for the recently developed estimators for staggered difference-in-differences to produce statistical tests that achieve 80% power. Further, conditional on statistical significance, when the intervention generates weak effects, estimators recover the wrong sign in approximately 10% of the simulations and overestimate the true effect by several hundred percent on average. We use data on publicly traded firms to investigate which sample size is needed for a staggered difference-in-differences analysis to be informative. We find that depending on the dependent variable and effect size, even the most efficient estimators generally need more than 250 units to achieve reasonable power. We conclude with a discussion of how this type of ‘design analysis’ ought to be used by researchers before estimating staggered difference-in-differences models. We also discuss how power may under certain conditions be improved if a study is re-designed, e.g., by examining county-level outcomes with state-level interventions.