The impact of study guides on “matric” performance: Evidence from a randomised experiment

Most international literature on the impact of textbooks on educational achievement suggests that this is a relatively cost-effective intervention. However, recent experimental evidence from developing countries has called this into question, suggesting that resources alone are unlikely to impact on performance and that changes in school organisation, pedagogical methods or incentives facing teachers are more effective. South African studies, using observational data, typically show weak associations between achievement and additional resources, though in some studies textbooks emerge as an exception. Some argue that school management is a key mediating variable. This paper evaluates the impact of providing study guides to pupils shortly before their secondary school leaving examination (the “matric” exam). From a sampling frame of 318 schools in the Mpumalanga province, 79 schools were randomly selected to receive study guides, leaving 239 control schools. These study guides were developed by the National Department of Basic Education and distributed to treatment schools for four subjects – accounting, economics, geography and life sciences – resulting in four distinct treatments per school. The impact of the study guides was estimated using matric results from 2011 (baseline) and 2012 (endline). The accounting and economics guides did not have a significant impact on performance. However, the geography and life sciences guides improved scores in those subjects by approximately two percentage points. Treatment heterogeneity was apparent for geography where students in better-performing schools gained more from the guides than students in low-performing schools. This may relate to other studies suggesting that additional school resources matter conditionally upon overall school functionality, particularly management. A simulation indicated that distributing the geography and life science at scale could increase the overall matric pass rate by roughly one percentage point. A cost-benefit analysis calculating the standard deviations of impact on test scores per $100 spent indicates that this intervention is amongst the most cost-effective of educational interventions internationally that have been tested using randomised experiments. Possible reasons why the guides were effective in two subjects but not the other two are discussed.