IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp13598.html
   My bibliography  Save this paper

Some Children Left Behind: Variation in the Effects of an Educational Intervention

Author

Listed:
  • Buhl-Wiggers, Julie

    (Copenhagen Business School)

  • Kerwin, Jason

    (University of Minnesota)

  • Muñoz, Juan Sebastián

    (IÉSEG School of Management)

  • Smith, Jeffrey A.

    (University of Wisconsin-Madison)

  • Thornton, Rebecca L.

    (University of Illinois)

Abstract

We document substantial variation in the effects of a highly-effective literacy program in northern Uganda. The program increases test scores by 1.4 SDs on average, but standard statistical bounds show that the impact standard deviation exceeds 1.0 SD. This implies that the variation in effects across our students is wider than the spread of mean effects across all randomized evaluations of developing country education interventions in the literature. This very effective program does indeed leave some students behind. At the same time, we do not learn much from our analyses that attempt to determine which students benefit more or less from the program. We reject rank preservation, and the weaker assumption of stochastic increasingness leaves wide bounds on quantile-specific average treatment effects. Neither conventional nor machine-learning approaches to estimating systematic heterogeneity capture more than a small fraction of the variation in impacts given our available candidate moderators.

Suggested Citation

  • Buhl-Wiggers, Julie & Kerwin, Jason & Muñoz, Juan Sebastián & Smith, Jeffrey A. & Thornton, Rebecca L., 2020. "Some Children Left Behind: Variation in the Effects of an Educational Intervention," IZA Discussion Papers 13598, Institute of Labor Economics (IZA).
  • Handle: RePEc:iza:izadps:dp13598
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp13598.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    2. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukherji & Marc Shotland & Michael Walton, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India," NBER Working Papers 22746, National Bureau of Economic Research, Inc.
    3. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 487-535.
    4. Shuai Chen & Lu Tian & Tianxi Cai & Menggang Yu, 2017. "A general statistical framework for subgroup identification and comparative treatment scoring," Biometrics, The International Biometric Society, vol. 73(4), pages 1199-1209, December.
    5. David K. Evans & Anna Popova, 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews," The World Bank Research Observer, World Bank Group, vol. 31(2), pages 242-270.
    6. Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
    7. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    8. Kirabo Jackson & Alexey Makarin, 2018. "Can Online Off-the-Shelf Lessons Improve Student Outcomes? Evidence from a Field Experiment," American Economic Journal: Economic Policy, American Economic Association, vol. 10(3), pages 226-254, August.
    9. Glewwe, Paul (ed.), 2013. "Education Policy in Developing Countries," University of Chicago Press Economics Books, University of Chicago Press, number 9780226078687, February.
    10. Bitler, Marianne P. & Gelbach, Jonah B. & Hoynes, Hilary W., 2008. "Distributional impacts of the Self-Sufficiency Project," Journal of Public Economics, Elsevier, vol. 92(3-4), pages 748-765, April.
    11. Peter Boone & Ila Fazzio & Kameshwari Jandhyala & Chitra Jayanty & Gangadhar Jayanty & Simon Johnson & Vimala Ramachandran & Filipa Silva & Zhaoguo Zhan, 2014. "The Surprisingly Dire Situation of Children's Education in Rural West Africa: Results from the CREO Study in Guinea-Bissau (Comprehensive Review of Education Outcomes)," NBER Chapters, in: African Successes, Volume II: Human Capital, pages 255-280, National Bureau of Economic Research, Inc.
    12. Michael C. Knaus & Michael Lechner & Anthony Strittmatter, 2022. "Heterogeneous Employment Effects of Job Search Programs: A Machine Learning Approach," Journal of Human Resources, University of Wisconsin Press, vol. 57(2), pages 597-636.
    13. Lu Tian & Ash A. Alizadeh & Andrew J. Gentles & Robert Tibshirani, 2014. "A Simple Method for Estimating Interactions Between a Treatment and a Large Number of Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1517-1532, December.
    14. Krishnaratne, Shari & White, Howard, 2013. "Quality education for all children? What works in education in developing countries," 3ie Publications 0000-0, International Initiative for Impact Evaluation (3ie).
    15. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    16. Jaegeum Lim & Jonathan Meer, 2017. "The Impact of Teacher–Student Gender Matches: Random Assignment Evidence from South Korea," Journal of Human Resources, University of Wisconsin Press, vol. 52(4), pages 979-997.
    17. Delavallade,Clara Anne & Griffith,Alan & Thornton,Rebecca Lynn, 2019. "Effects of a Multi-Faceted Education Program on Enrollment, Equity, Learning, and School Management : Evidence from India," Policy Research Working Paper Series 9081, The World Bank.
    18. Abhijit Banerjee & Rukmini Banerji & Esther Duflo, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level†in India," Working Papers id:11419, eSocialSciences.
    19. Sylvie Moulin & Michael Kremer & Paul Glewwe, 2009. "Many Children Left Behind? Textbooks and Test Scores in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 112-135, January.
    20. Jay Bhattacharya & Azeem M. Shaikh & Edward Vytlacil, 2008. "Treatment Effect Bounds under Monotonicity Assumptions: An Application to Swan-Ganz Catheterization," American Economic Review, American Economic Association, vol. 98(2), pages 351-356, May.
    21. Michael J. Weiss & Howard S. Bloom & Thomas Brock, 2014. "A Conceptual Framework For Studying The Sources Of Variation In Program Effects," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(3), pages 778-808, June.
    22. Bitler, Marianne P. & Gelbach, Jonah B. & Hoynes, Hilary W., 2008. "Distributional impacts of the Self-Sufficiency Project," Journal of Public Economics, Elsevier, vol. 92(3-4), pages 748-765, April.
    23. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 114(2), pages 533-575.
    24. Black, Dan A. & Smith, J.A.Jeffrey A., 2004. "How robust is the evidence on the effects of college quality? Evidence from matching," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 99-124.
    25. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2019. "The Economics of Artificial Intelligence: An Agenda," NBER Books, National Bureau of Economic Research, Inc, number agra-1, January.
    26. Alejandro J. Ganimian & Richard J. Murnane, 2014. "Improving Educational Outcomes in Developing Countries: Lessons from Rigorous Impact Evaluations," NBER Working Papers 20284, National Bureau of Economic Research, Inc.
    27. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    28. Jacobus Cilliers & Brahm Fleisch & Cas Prinsloo & Stephen Taylor, 2020. "How to Improve Teaching Practice?: An Experimental Comparison of Centralized Training and In-Classroom Coaching," Journal of Human Resources, University of Wisconsin Press, vol. 55(3), pages 926-962.
    29. Tessa Bold & Deon Filmer & Gayle Martin & Ezequiel Molina & Brian Stacy & Christophe Rockmore & Jakob Svensson & Waly Wane, 2017. "Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 185-204, Fall.
    30. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    31. Fan, Yanqin & Park, Sang Soo, 2010. "Sharp Bounds On The Distribution Of Treatment Effects And Their Statistical Inference," Econometric Theory, Cambridge University Press, vol. 26(3), pages 931-951, June.
    32. Piper, Benjamin & Zuilkowski, Stephanie Simmons & Kwayumba, Dunston & Oyanga, Arbogast, 2018. "Examining the secondary effects of mother-tongue literacy instruction in Kenya: Impacts on student learning in English, Kiswahili, and mathematics," International Journal of Educational Development, Elsevier, vol. 59(C), pages 110-127.
    33. World Bank, 2018. "World Development Report 2018 [Rapport sur le développement dans le monde 2018]," World Bank Publications - Books, The World Bank Group, number 28340, December.
    34. Jonathan M.V. Davis & Sara B. Heller, 2017. "Using Causal Forests to Predict Treatment Heterogeneity: An Application to Summer Jobs," American Economic Review, American Economic Association, vol. 107(5), pages 546-550, May.
    35. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    36. Eric A. Hanushek & Steven G. Rivkin, 2010. "Generalizations about Using Value-Added Measures of Teacher Quality," American Economic Review, American Economic Association, vol. 100(2), pages 267-271, May.
    37. Agrawal, Ajay & Gans, Joshua & Goldfarb, Avi (ed.), 2019. "The Economics of Artificial Intelligence," National Bureau of Economic Research Books, University of Chicago Press, number 9780226613338, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Piper, Benjamin & Simmons Zuilkowski, Stephanie & Dubeck, Margaret & Jepkemei, Evelyn & King, Simon J., 2018. "Identifying the essential ingredients to literacy and numeracy improvement: Teacher professional development and coaching, student textbooks, and structured teachers’ guides," World Development, Elsevier, vol. 106(C), pages 324-336.
    2. Blimpo, Moussa P. & Pugatch, Todd, 2021. "Entrepreneurship education and teacher training in Rwanda," Journal of Development Economics, Elsevier, vol. 149(C).
    3. Francesca Marchetta & Tom Dilly, 2019. "Supporting Education in Africa: Opportunities and Challenges for an Impact Investor," Working Papers hal-02288103, HAL.
    4. Rodriguez-Segura, Daniel, 2022. "A closer look at reading comprehension: Experimental evidence from Guatemala," International Journal of Educational Development, Elsevier, vol. 93(C).
    5. Eiji Koazuka, 2018. "Enlightening Communities and Parents for Improving Student Learning Evidence from Randomized Experiment in Niger," Working Papers 166, JICA Research Institute.
    6. Angrist, Noam & de Barros, Andreas & Bhula, Radhika & Chakera, Shiraz & Cummiskey, Chris & DeStefano, Joseph & Floretta, John & Kaffenberger, Michelle & Piper, Benjamin & Stern, Jonathan, 2021. "Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery," International Journal of Educational Development, Elsevier, vol. 84(C).
    7. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    8. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    9. Sawada Yasuyuki & Mahmud Minhaj & Seki Mai & Le An & Kawarazaki Hikaru, 2017. "Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh," Working Papers 156, JICA Research Institute.
    10. Yasuyuki Sawada & Minhaj Mahmud & Mai Seki & An Le & Hikaru Kawarazaki, 2019. "Fighting the Learning Crisis in Developing Countries: A Randomized Experiment of Self-Learning at the Right Level," CIRJE F-Series CIRJE-F-1127, CIRJE, Faculty of Economics, University of Tokyo.
    11. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    12. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1627-1673.
    13. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    14. Cilliers, Jacobus & Fleisch, Brahm & Kotze, Janeli & Mohohlwane, Nompumelelo & Taylor, Stephen & Thulare, Tsegofatso, 2022. "Can virtual replace in-person coaching? Experimental evidence on teacher professional development and student learning," Journal of Development Economics, Elsevier, vol. 155(C).
    15. Lechner, Michael, 2018. "Modified Causal Forests for Estimating Heterogeneous Causal Effects," IZA Discussion Papers 12040, Institute of Labor Economics (IZA).
    16. Aymo Brunetti & Konstantin Büchel & Martina Jakob & Ben Jann & Daniel Steffen, 2021. "Inadequate Teacher Content Knowledge and What to Do About It: Evidence from El Salvador," University of Bern Social Sciences Working Papers 41, University of Bern, Department of Social Sciences.
    17. Djebbari, Habiba & Smith, Jeffrey, 2008. "Heterogeneous impacts in PROGRESA," Journal of Econometrics, Elsevier, vol. 145(1-2), pages 64-80, July.
    18. Jules Gazeaud & Claire Ricard, 2021. "Conditional cash transfers and the learning crisis: evidence from Tayssir scale-up in Morocco," NOVAFRICA Working Paper Series wp2102, Universidade Nova de Lisboa, Nova School of Business and Economics, NOVAFRICA.
    19. Michael C. Knaus & Michael Lechner & Anthony Strittmatter, 2018. "Machine Learning Estimation of Heterogeneous Causal Effects: Empirical Monte Carlo Evidence," Papers 1810.13237, arXiv.org, revised Dec 2018.
    20. Fazzio, Ila & Eble, Alex & Lumsdaine, Robin L. & Boone, Peter & Bouy, Baboucarr & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Johnson, Simon & Silva, Ana Filipa, 2021. "Large learning gains in pockets of extreme poverty: Experimental evidence from Guinea Bissau," Journal of Public Economics, Elsevier, vol. 199(C).

    More about this item

    Keywords

    essential heterogeneity; heterogeneous treatment effects; education;
    All these keywords.

    JEL classification:

    • I25 - Health, Education, and Welfare - - Education - - - Education and Economic Development
    • I26 - Health, Education, and Welfare - - Education - - - Returns to Education

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp13598. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.