Beyond Russell reconstitution: A re-examination of methodologies for natural experiments

We develop a framework for screening and validating empirical designs in natural experiments. As an illustration of the framework, we establish a common testing ground for three popular research designs – instrumental variables (IV), fuzzy regression discontinuity (FRD), and difference-in-differences (DiD) – that exploit the annual Russell 1000/2000 Indexes reconstitution. Of the three designs, we find that only the IV approach spuriously detects large and statistically significant “effects” that by construction should be immaterial and statistically indistinguishable from zero. We advocate the use of simulation evidence to support key identifying assumptions for future research designs based on natural experiments.