Putting AI to the Test: Evidence from a Large-Scale RCT in Rural China

The emergence of artificial intelligence (AI) has heightened interest in personalizing computer assisted learning (CAL) programs to tailor their instruction to individual students. Despite the proliferation of AI-driven CAL programs, evidence for their effectiveness remains limited. We present findings from a large-scale field experiment in rural China examining whether an AI-driven personalized CAL program improves student achievement. We randomly assign 8,647 students from 315 primary school classes to one of three treatment arms: (i) AI-CAL, (ii) non-personalized CAL (active control), and (iii) non-CAL educational activities (pure control). Results indicate that AI-CAL does not significantly improve achievement, with estimates precise enough to rule out non-trivial positive effects. This finding holds across the difficulty of assessment items and across the baseline achievement of students. The finding that R-CAL modestly benefits students in the middle ability tercile while AI-CAL shows no impact on students in any ability tercile suggests caution when projecting the promise of scaling adaptive AI-driven educational technologies in under-resourced settings.