Using lotteries to evaluate schools of choice: Evidence from a national study of charter schools

This paper draws on data and experiences observing and analyzing school lotteries from the National Evaluation of Charter School Impacts (Gleason et al., 2010) to describe the challenges associated with lottery-based research. In that study, covering 36 charter middle schools in 15 states, we found that charter schools did not affect student achievement or behavior on average, although there was substantial variation across schools. In this paper, we discuss the prevalence of oversubscribed charter schools at the time the study was conducted (the 2005–2006 and 2006–2007 school years), which was lower than commonly reported. We then describe how the sample of schools that participated in the study compared to all other charter middle schools nationwide, to provide some insight into the generalizabilty of findings from lottery-based studies. In general, oversubscribed charter schools were more likely to be located in urban areas and serve a higher-achieving population of students than those without excess demand. We also describe common features of school lotteries and waitlists, and examine implications of these features for a school's ability to support a lottery-based study. Finally, we summarize lessons learned for conducting lottery-based research on charter schools, drawing on our observations of the schools’ lotteries and analysis of the data from these lotteries.