Random Placement but Real Bias

Many studies exploit the random placement of individuals into groups such as schools or regions to estimate the effects of group-level variables on these individuals. Assuming a simple data generating process, we show that the typical estimate contains three components: the causal effect of interest, “multiple-treatment bias” (MTB), and “mobility bias” (MB). The extent of these biases depends on the interrelations of group-level variables and onward mobility. We develop a checklist that can be used to assess the relevance of the biases based on observable quantities. We apply this framework to novel administrative data on randomly placed refugees in Germany and confirm empirically that MTB and MB cannot be ignored. The biases can even switch the signs of estimates of popular group-level variables, despite random placement. We discuss implications for the literature and alternative “ideal experiments".