New Evidence and Design Considerations for Repeated Measure Experiments in Survey Research

An influential study in the American Political Science Review by Clifford, Sheagley, and Piston (2021) finds that including pre-treatment measures of outcome variables in survey experiments does not bias treatment effect estimates and greatly improves precision, prompting many researchers to adopt repeated measure designs. Though promising, the 2021 study remains the only empirical investigation to date to support this broad shift in experimental practice. In a large-scale partial replication, we experimentally manipulate the design of six classic experiments in political science and field all six experiments in three separate samples of U.S. adults (N = 13,163) to re-examine the central claim that repeated measure designs do not bias treatment effects. We also provide three extensions on additional design considerations regarding within-subject experimental designs, the relative separation of repeated measures within single surveys, and respondent characteristics in probability versus non-probability samples. In contrast to the original study, we find evidence of attenuation of treatment effects in repeated measure designs. However, this average attenuation bias is sufficiently small that we largely affirm the original authors' recommendation to prefer repeated measure designs in most research applications, because the large gains to statistical precision will typically produce a more accurate estimate ATE in expectation.