Sample Attrition Bias in Randomized Experiments : A Table of Two Surveys

The randomized trial literature has helped to renew the field of microeconometric policy evaluation byemphasizing identification issues raised by endogenous program participation. Measurement andattrition issues have perhaps received less attention. This paper analyzes the dramatic impact of sampleattrition in a large job search experiment. We take advantage of two independent surveys on the sameinitial sample of 8,000 persons. The first one is a long telephone survey that had a strikingly low andunbalanced response rate of about 50%. The second one is a combination of administrative data and ashort telephone survey targeted at those leaving the unemployment registers; this enriched data sourcehas a balanced and much higher response rate (about 80%). With naive estimates that neglect nonresponses, these two sources yield puzzlingly different results.Using the enriched administrative data as benchmark, we find evidence that estimates from the longtelephone survey lack external and internal validity. We turn to existing methods to bound the effectsin the presence of sample selection; we extend them to the context of randomization with imperfectcompliance. The bounds obtained from the two surveys are compatible but those from the longtelephone survey are somewhat uninformative. We conclude on the consequences for data collectionstrategies.