Experimental evaluation of mail questionnaires in a probability sample on victimization

Experiments to evaluate questionnaires or methods for data collection are often conducted in the context of a probability sample that collects data from various primary sampling units or sites. Statistics used to evaluate treatment effects for these experiments have different interpretations and variances in different inferential frameworks. We discuss four frameworks for inference about treatment effects: inference to the finite population, to a superpopulation of future sites, to the mean of the site treatment effects and to individual respondents. For each framework, we consider the parameters of interest, the properties of statistics used to estimate those parameters, and the optimal design for the experiment. We consider the four inferential frameworks for an experiment conducted on a mail survey measuring criminal victimization and community attitudes towards law enforcement. The paper concludes with a discussion of how the frameworks could be applied to other experiments.