Scaling Behavioral Interventions in the Presence of Spillover

Randomized Controlled Trials (RCTs) are increasingly relied upon by policymakers as part of efforts to incorporate evidence into the policymaking process, a movement known as evidence-based policymaking, or EBPM. Testing possible policy interventions via RCTs before full rollout is commonly thought to be the gold standard of evidence in the EBPM process. However, real-world policy changes do not always scale up as expected. Even large-N RCTs targeting a random sample of policy beneficiaries do not capture the influence of social networks and risk missing consequential spillover effects. We illustrate this issue by assessing the efficacy of monetary incentives to increase COVID-19 vaccination in an RCT over the entire population of a medium-sized European town (~40,000 residents). We use administrative vaccination data as our primary outcome. Since the entire population was randomized, we are able to estimate spillover effects within households. There were significant negative spillover effects on booster vaccinations that we attribute to a displacement effect, potentially driven by long lines at the vaccination events. Our results illustrate that using a population-level RCT to test whether a policy scales can help avoid costly, ineffective, or even counterproductive policy outcomes.