Publication bias inflates estimates of the effectiveness of common educational interventions

While there is a large body of evidence on the effectiveness of educational interventions, the robustness of this evidence has not been systematically assessed. Here we conduct a preregistered systematic review, quality appraisal, and meta-analysis of all available estimates of the effects of four common supplementary learning interventions—tutoring, summer programs, afterschool programs, and extended instruction time—on children’s learning in mathematics and reading. We use multiple diagnostic techniques to assess the risk of bias of individual effect size estimates and to test for publication bias in the evidence base on supplementary learning interventions as a whole. We find that about half of all effect size estimates are at serious or critical risk of bias, with confounding and deviation from intended treatment being the most prominent risk-of-bias domains. We find substantial publication bias in the existing evidence base, indicated by a marked discontinuity in the distribution of test statistics at the conventional significance threshold and a significant association between magnitude and precision of effect size estimates. We combine five bias-correction techniques to estimate bias-adjusted effects for the four intervention types. Correcting for publication bias substantially reduces effect size estimates for tutoring, summer programs, and extended instruction time and renders estimates for afterschool programs statistically insignificant. After correction, tutoring is the only intervention that consistently yields positive and statistically significant effects. Our findings suggest that the existing evidence base substantially overestimates the effectiveness of commonly implemented supplementary learning interventions.