The Illusion of Comparability Among Standardised Effect Sizes: Why Education Evaluations Should Report Raw Effects

Policymakers, practitioners, and donors rely on standardised effect sizes to compare education programmes and allocate resources. Yet in early-grade reading interventions in low- and middle-income countries, these metrics reflect differences in measurement tools, sample composition, and scaling choices as much as genuine learning gains. Drawing on data from 197 studies, we find that a single additional word per minute of reading fluency corresponds to anywhere from 0.03 to 0.55 standard deviations across studies. For tests designed by researchers—the most common type of test in this literature and the hardest to compare—a single correct response on a reading assessment can result in a standardised gain anywhere from 0.08 to 0.80 standard deviations. Converting raw to standardised effects also reorders which programmes appear most effective. Fewer than one in four papers (23 percent) present both standardised and raw effects, leaving readers unable to judge specifically what children can do differently as a result of the programme and whether that change matters educationally. We propose that researchers report raw effects and reference distributions alongside standardised estimates, that work on reading benchmarks be extended across more languages and countries, and that shared measurement frameworks be developed faster. Together, these reporting changes can improve interpretation for policymakers and allow evidence syntheses to explicitly model differences in test design and sample composition across studies. Each of these changes would help the field show what children actually gained from a programme, not just how large the effect appears.