IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/29459.html
   My bibliography  Save this paper

Some Children Left Behind: Variation in the Effects of an Educational Intervention

Author

Listed:
  • Julie Buhl-Wiggers
  • Jason T. Kerwin
  • Juan S. Muñoz-Morales
  • Jeffrey A. Smith
  • Rebecca Thornton

Abstract

We document substantial variation in the effects of a highly-effective literacy pro-gram in northern Uganda. The program increases test scores by 1.40 SDs on average, but standard statistical bounds show that the impact standard deviation exceeds 1.0SD. This implies that the variation in effects across our students is wider than the spread of mean effects across all randomized evaluations of developing country education interventions in the literature. This very effective program does indeed leave some students behind. At the same time, we do not learn much from our analyses that attempt to determine which students benefit more or less from the program. We reject rank preservation, and the weaker assumption of stochastic increasingness leaves wide bounds on quantile-specific average treatment effects. Neither conventional nor machine-learning approaches to estimating systematic heterogeneity capture more than a small fraction of the variation in impacts given our available candidate moderators.

Suggested Citation

  • Julie Buhl-Wiggers & Jason T. Kerwin & Juan S. Muñoz-Morales & Jeffrey A. Smith & Rebecca Thornton, 2021. "Some Children Left Behind: Variation in the Effects of an Educational Intervention," NBER Working Papers 29459, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:29459
    Note: DEV ED
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w29459.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
    2. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    3. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukherji & Marc Shotland & Michael Walton, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India," NBER Working Papers 22746, National Bureau of Economic Research, Inc.
    4. Black, Dan A. & Smith, J.A.Jeffrey A., 2004. "How robust is the evidence on the effects of college quality? Evidence from matching," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 99-124.
    5. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2019. "The Economics of Artificial Intelligence: An Agenda," NBER Books, National Bureau of Economic Research, Inc, number agra-1.
    6. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    7. Shuai Chen & Lu Tian & Tianxi Cai & Menggang Yu, 2017. "A general statistical framework for subgroup identification and comparative treatment scoring," Biometrics, The International Biometric Society, vol. 73(4), pages 1199-1209, December.
    8. David K. Evans & Anna Popova, 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews," The World Bank Research Observer, World Bank, vol. 31(2), pages 242-270.
    9. Alejandro J. Ganimian & Richard J. Murnane, 2014. "Improving Educational Outcomes in Developing Countries: Lessons from Rigorous Impact Evaluations," NBER Working Papers 20284, National Bureau of Economic Research, Inc.
    10. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    11. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    12. Kirabo Jackson & Alexey Makarin, 2018. "Can Online Off-the-Shelf Lessons Improve Student Outcomes? Evidence from a Field Experiment," American Economic Journal: Economic Policy, American Economic Association, vol. 10(3), pages 226-254, August.
    13. Glewwe, Paul (ed.), 2013. "Education Policy in Developing Countries," University of Chicago Press Economics Books, University of Chicago Press, number 9780226078687.
    14. Sylvie Moulin & Michael Kremer & Paul Glewwe, 2009. "Many Children Left Behind? Textbooks and Test Scores in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 112-135, January.
    15. Bitler, Marianne P. & Gelbach, Jonah B. & Hoynes, Hilary W., 2008. "Distributional impacts of the Self-Sufficiency Project," Journal of Public Economics, Elsevier, vol. 92(3-4), pages 748-765, April.
    16. Jacobus Cilliers & Brahm Fleisch & Cas Prinsloo & Stephen Taylor, 2020. "How to Improve Teaching Practice?: An Experimental Comparison of Centralized Training and In-Classroom Coaching," Journal of Human Resources, University of Wisconsin Press, vol. 55(3), pages 926-962.
    17. Tessa Bold & Deon Filmer & Gayle Martin & Ezequiel Molina & Brian Stacy & Christophe Rockmore & Jakob Svensson & Waly Wane, 2017. "Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 185-204, Fall.
    18. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    19. Peter Boone & Ila Fazzio & Kameshwari Jandhyala & Chitra Jayanty & Gangadhar Jayanty & Simon Johnson & Vimala Ramachandran & Filipa Silva & Zhaoguo Zhan, 2014. "The Surprisingly Dire Situation of Children's Education in Rural West Africa: Results from the CREO Study in Guinea-Bissau (Comprehensive Review of Education Outcomes)," NBER Chapters, in: African Successes, Volume II: Human Capital, pages 255-280, National Bureau of Economic Research, Inc.
    20. James J. Heckman & Sergio Urzua & Edward Vytlacil, 2006. "Understanding Instrumental Variables in Models with Essential Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 88(3), pages 389-432, August.
    21. Abhijit Banerjee & Rukmini Banerji & Esther Duflo, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level†in India," Working Papers id:11419, eSocialSciences.
    22. Michael C. Knaus & Michael Lechner & Anthony Strittmatter, 2022. "Heterogeneous Employment Effects of Job Search Programs: A Machine Learning Approach," Journal of Human Resources, University of Wisconsin Press, vol. 57(2), pages 597-636.
    23. Fan, Yanqin & Park, Sang Soo, 2010. "Sharp Bounds On The Distribution Of Treatment Effects And Their Statistical Inference," Econometric Theory, Cambridge University Press, vol. 26(3), pages 931-951, June.
    24. Piper, Benjamin & Zuilkowski, Stephanie Simmons & Kwayumba, Dunston & Oyanga, Arbogast, 2018. "Examining the secondary effects of mother-tongue literacy instruction in Kenya: Impacts on student learning in English, Kiswahili, and mathematics," International Journal of Educational Development, Elsevier, vol. 59(C), pages 110-127.
    25. Lu Tian & Ash A. Alizadeh & Andrew J. Gentles & Robert Tibshirani, 2014. "A Simple Method for Estimating Interactions Between a Treatment and a Large Number of Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1517-1532, December.
    26. World Bank, 2018. "World Development Report 2018 [Rapport sur le développement dans le monde 2018]," World Bank Publications - Books, The World Bank Group, number 28340.
    27. Krishnaratne, Shari & White, Howard, 2013. "Quality education for all children? What works in education in developing countries," 3ie Publications 0000-0, International Initiative for Impact Evaluation (3ie).
    28. Jonathan M.V. Davis & Sara B. Heller, 2017. "Using Causal Forests to Predict Treatment Heterogeneity: An Application to Summer Jobs," American Economic Review, American Economic Association, vol. 107(5), pages 546-550, May.
    29. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    30. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    31. Jaegeum Lim & Jonathan Meer, 2017. "The Impact of Teacher–Student Gender Matches: Random Assignment Evidence from South Korea," Journal of Human Resources, University of Wisconsin Press, vol. 52(4), pages 979-997.
    32. Eric A. Hanushek & Steven G. Rivkin, 2010. "Generalizations about Using Value-Added Measures of Teacher Quality," American Economic Review, American Economic Association, vol. 100(2), pages 267-271, May.
    33. Delavallade,Clara Anne & Griffith,Alan & Thornton,Rebecca Lynn, 2019. "Effects of a Multi-Faceted Education Program on Enrollment, Equity, Learning, and School Management : Evidence from India," Policy Research Working Paper Series 9081, The World Bank.
    34. Agrawal, Ajay & Gans, Joshua & Goldfarb, Avi (ed.), 2019. "The Economics of Artificial Intelligence," National Bureau of Economic Research Books, University of Chicago Press, number 9780226613338.
    35. Jay Bhattacharya & Azeem M. Shaikh & Edward Vytlacil, 2008. "Treatment Effect Bounds under Monotonicity Assumptions: An Application to Swan-Ganz Catheterization," American Economic Review, American Economic Association, vol. 98(2), pages 351-356, May.
    36. Michael J. Weiss & Howard S. Bloom & Thomas Brock, 2014. "A Conceptual Framework For Studying The Sources Of Variation In Program Effects," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(3), pages 778-808, June.
    37. Bitler, Marianne P. & Gelbach, Jonah B. & Hoynes, Hilary W., 2008. "Distributional impacts of the Self-Sufficiency Project," Journal of Public Economics, Elsevier, vol. 92(3-4), pages 748-765, April.
    38. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 533-575.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marianne Simonsen & Lars Skipper, 2024. "Healthy at Work? Evidence from a Social Experimental Evaluation of a Firm-Based Wellness Program," CESifo Working Paper Series 11209, CESifo.
    2. Li, Yanan & Sunder, Naveen, 2024. "Distributional effects of education on mental health," Labour Economics, Elsevier, vol. 88(C).
    3. Muhammad Meki & Simon Quinn, 2024. "Microfinance: an overview," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 40(1), pages 1-7.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Piper, Benjamin & Simmons Zuilkowski, Stephanie & Dubeck, Margaret & Jepkemei, Evelyn & King, Simon J., 2018. "Identifying the essential ingredients to literacy and numeracy improvement: Teacher professional development and coaching, student textbooks, and structured teachers’ guides," World Development, Elsevier, vol. 106(C), pages 324-336.
    2. Francesca Marchetta & Tom Dilly, 2019. "Supporting Education in Africa: Opportunities and Challenges for an Impact Investor," Working Papers hal-02288103, HAL.
    3. Blimpo, Moussa P. & Pugatch, Todd, 2021. "Entrepreneurship education and teacher training in Rwanda," Journal of Development Economics, Elsevier, vol. 149(C).
    4. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    5. Rodriguez-Segura, Daniel, 2022. "A closer look at reading comprehension: Experimental evidence from Guatemala," International Journal of Educational Development, Elsevier, vol. 93(C).
    6. Eiji Koazuka, 2018. "Enlightening Communities and Parents for Improving Student Learning Evidence from Randomized Experiment in Niger," Working Papers 166, JICA Research Institute.
    7. Angrist, Noam & de Barros, Andreas & Bhula, Radhika & Chakera, Shiraz & Cummiskey, Chris & DeStefano, Joseph & Floretta, John & Kaffenberger, Michelle & Piper, Benjamin & Stern, Jonathan, 2021. "Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery," International Journal of Educational Development, Elsevier, vol. 84(C).
    8. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    9. Sawada Yasuyuki & Mahmud Minhaj & Seki Mai & Le An & Kawarazaki Hikaru, 2017. "Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh," Working Papers 156, JICA Research Institute.
    10. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    11. Yasuyuki Sawada & Minhaj Mahmud & Mai Seki & An Le & Hikaru Kawarazaki, 2019. "Fighting the Learning Crisis in Developing Countries: A Randomized Experiment of Self-Learning at the Right Level," CIRJE F-Series CIRJE-F-1127, CIRJE, Faculty of Economics, University of Tokyo.
    12. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    13. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(3), pages 1627-1673.
    14. Lee, Jinhyun, 2013. "Sharp Bounds on Heterogeneous Individual Treatment Responses," SIRE Discussion Papers 2013-89, Scottish Institute for Research in Economics (SIRE).
    15. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    16. Jinhyun Lee, 2013. "Sharp Bounds on Heterogeneous Individual Treatment Responses," Discussion Paper Series, School of Economics and Finance 201310, School of Economics and Finance, University of St Andrews.
    17. Cilliers, Jacobus & Fleisch, Brahm & Kotze, Janeli & Mohohlwane, Nompumelelo & Taylor, Stephen & Thulare, Tsegofatso, 2022. "Can virtual replace in-person coaching? Experimental evidence on teacher professional development and student learning," Journal of Development Economics, Elsevier, vol. 155(C).
    18. Lechner, Michael, 2018. "Modified Causal Forests for Estimating Heterogeneous Causal Effects," IZA Discussion Papers 12040, Institute of Labor Economics (IZA).
    19. Aymo Brunetti & Konstantin Büchel & Martina Jakob & Ben Jann & Daniel Steffen, 2021. "Inadequate Teacher Content Knowledge and What to Do About It: Evidence from El Salvador," University of Bern Social Sciences Working Papers 41, University of Bern, Department of Social Sciences.
    20. Djebbari, Habiba & Smith, Jeffrey, 2008. "Heterogeneous impacts in PROGRESA," Journal of Econometrics, Elsevier, vol. 145(1-2), pages 64-80, July.

    More about this item

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I25 - Health, Education, and Welfare - - Education - - - Education and Economic Development
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:29459. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.