Strong control of the familywise error rate in observational studies that discover effect modification by exploratory methods

An effect modifier is a pretreatment covariate that affects the magnitude of the treatment effect or its stability. When there is effect modification, an overall test that ignores an effect modifier may be more sensitive to unmeasured bias than a test that combines results from subgroups defined by the effect modifier. If there is effect modification, one would like to identify specific subgroups for which there is evidence of effect that is insensitive to small or moderate biases. In this paper, we propose an exploratory method for discovering effect modification, and combine it with a confirmatory method of simultaneous inference that strongly controls the familywise error rate in a sensitivity analysis, despite the fact that the groups being compared are defined empirically. A new form of matching, strength-$k$ matching, permits a search through more than $k$ covariates for effect modifiers, in such a way that no pairs are lost, provided that at most $k$ covariates are selected to group the pairs. In a strength-$k$ match, each set of $k$ covariates is exactly balanced, although a set of more than $k$ covariates may exhibit imbalance. We apply the proposed method to study the effects of the earthquake that struck Chile in 2010.