IDEAS home Printed from https://ideas.repec.org/a/tpr/restat/v103y2021i2p251-264.html
   My bibliography  Save this article

Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures

Author

Listed:
  • Jason T. Kerwin

    (Department of Applied Economics, University of Minnesota)

  • Rebecca L. Thornton

    (Department of Economics, University of Illinois)

Abstract

This paper demonstrates the acute sensitivity of education program effectiveness to the choices of inputs and outcome measures, using a randomized evaluation of a mother-tongue literacy program. The program raises reading scores by 0.64 SD and writing scores by 0.45 SD. A reduced-cost version instead yields statistically insignificant reading gains and some large negative effects (−0.33 SDs) on advanced writing. We combine a conceptual model of education production with detailed classroom observations to examine the mechanisms driving the results; we show they could be driven by the program initially lowering productivity before raising it, and potentially by missing complementary inputs in the reduced-cost version.

Suggested Citation

  • Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
  • Handle: RePEc:tpr:restat:v:103:y:2021:i:2:p:251-264
    DOI: 10.1162/rest_a_00911
    as

    Download full text from publisher

    File URL: https://doi.org/10.1162/rest_a_00911
    Download Restriction: Access to PDF is restricted to subscribers.

    File URL: https://libkey.io/10.1162/rest_a_00911?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, Oxford University Press, vol. 122(3), pages 1235-1264.
    3. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    4. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    5. Jens Ludwig & Jeffrey R. Kling & Sendhil Mullainathan, 2011. "Mechanism Experiments and Policy Evaluations," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 17-38, Summer.
    6. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, Oxford University Press, vol. 130(3), pages 1117-1165.
    7. Pritchett, Lant & Filmer, Deon, 1999. "What education production functions really show: a positive theory of education expenditures," Economics of Education Review, Elsevier, vol. 18(2), pages 223-239, April.
    8. David K. Evans & Anna Popova, 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews," World Bank Research Observer, World Bank Group, vol. 31(2), pages 242-270.
    9. Friedman, Jerome H. & Hastie, Trevor & Tibshirani, Rob, 2010. "Regularization Paths for Generalized Linear Models via Coordinate Descent," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 33(i01).
    10. Karthik Muralidharan & Venkatesh Sundararaman, 2013. "Contract Teachers: Experimental Evidence from India," NBER Working Papers 19440, National Bureau of Economic Research, Inc.
    11. Hainmueller, Jens & Hazlett, Chad, 2014. "Kernel Regularized Least Squares: Reducing Misspecification Bias with a Flexible and Interpretable Machine Learning Approach," Political Analysis, Cambridge University Press, vol. 22(2), pages 143-168, April.
    12. Daniel O. Gilligan & Naureen Karachiwalla & Ibrahim Kasirye & Adrienne M. Lucas & Derek Neal, 2022. "Educator Incentives and Educational Triage in Rural Primary Schools," Journal of Human Resources, University of Wisconsin Press, vol. 57(1), pages 79-111.
    13. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    14. Glewwe, Paul & Kremer, Michael & Moulin, Sylvie & Zitzewitz, Eric, 2004. "Retrospective vs. prospective analyses of school inputs: the case of flip charts in Kenya," Journal of Development Economics, Elsevier, vol. 74(1), pages 251-268, June.
    15. Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
    16. Melody M. Chao & Rajeev Dehejia & Anirban Mukhopadhyay & Sujata Visaria, 2015. "Unintended Negative Consequences of Rewards for Student Attendance: Results from a Field Experiment in Indian Classrooms," HKUST IEMS Working Paper Series 2015-22, HKUST Institute for Emerging Market Studies, revised Apr 2015.
    17. David S. Lee, 2009. "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects," Review of Economic Studies, Oxford University Press, vol. 76(3), pages 1071-1102.
    18. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LP, vol. 19(1), pages 4-60, March.
    19. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1627-1673.
    20. Jere R. Behrman & Susan W. Parker & Petra E. Todd & Kenneth I. Wolpin, 2015. "Aligning Learning Incentives of Students and Teachers: Results from a Social Experiment in Mexican High Schools," Journal of Political Economy, University of Chicago Press, vol. 123(2), pages 325-364.
    21. Nadel, Sara & Pritchett, Lant, 2016. "Searching for the Devil in the Details: Learning about Development Program Design," Working Paper Series rwp16-041, Harvard University, John F. Kennedy School of Government.
    22. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    23. Acharya, Avidit & Blackwell, Matthew & Sen, Maya, 2016. "Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects," American Political Science Review, Cambridge University Press, vol. 110(3), pages 512-529, August.
    24. Eva Vivalt, 2020. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    25. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    26. Popova,Anna & Evans,David & Arancibia,Violeta, 2016. "Training teachers on the job : what works and how to measure it," Policy Research Working Paper Series 7834, The World Bank.
    27. Piper, Benjamin & Simmons Zuilkowski, Stephanie & Dubeck, Margaret & Jepkemei, Evelyn & King, Simon J., 2018. "Identifying the essential ingredients to literacy and numeracy improvement: Teacher professional development and coaching, student textbooks, and structured teachers’ guides," World Development, Elsevier, vol. 106(C), pages 324-336.
    28. Dan A. Black & Jeffrey A. Smith, 2006. "Estimating the Returns to College Quality with Multiple Proxies for Quality," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 701-728, July.
    29. Sara Nadel and Lant Pritchett, 2016. "Searching for the Devil in the Details: Learning about Development Program Design," Working Papers 434, Center for Global Development.
    30. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    31. Jacobus Cilliers & Brahm Fleisch & Cas Prinsloo & Stephen Taylor, 2018. "How to improve teaching practice? Experimental comparison of centralized training and in-classroom coaching," Working Papers 15/2018, Stellenbosch University, Department of Economics.
    32. Wilbur Townsend, 2017. "ELASTICREGRESS: Stata module to perform elastic net regression, lasso regression, ridge regression," Statistical Software Components S458397, Boston College Department of Economics, revised 16 Apr 2018.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    2. Buhl-Wiggers, Julie & Kerwin, Jason & Muñoz, Juan Sebastián & Smith, Jeffrey A. & Thornton, Rebecca L., 2020. "Some Children Left Behind: Variation in the Effects of an Educational Intervention," IZA Discussion Papers 13598, Institute of Labor Economics (IZA).
    3. Daniel O. Gilligan & Naureen Karachiwalla & Ibrahim Kasirye & Adrienne M. Lucas & Derek Neal, 2022. "Educator Incentives and Educational Triage in Rural Primary Schools," Journal of Human Resources, University of Wisconsin Press, vol. 57(1), pages 79-111.
    4. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers : The Impact of Student Access to Wikipedia," The Warwick Economics Research Paper Series (TWERPS) 1236, University of Warwick, Department of Economics.
    5. Aymo Brunetti & Konstantin Büchel & Martina Jakob & Ben Jann & Daniel Steffen, 2021. "Inadequate Teacher Content Knowledge and What to Do About It: Evidence from El Salvador," University of Bern Social Sciences Working Papers 41, University of Bern, Department of Social Sciences.
    6. Margaret Leighton, 2021. "Mother Tongue Reading Materials as a Bridge to Literacy," Discussion Paper Series, School of Economics and Finance 202101, School of Economics and Finance, University of St Andrews.
    7. Wesley Burnett, J. & Mothorpe, Christopher, 2021. "Human-induced earthquakes, risk salience, and housing values," Resource and Energy Economics, Elsevier, vol. 63(C).
    8. Annie Duflo & Jessica Kiessel & Adrienne Lucas, 2020. "Experimental Evidence on Alternative Policies to Increase Learning at Scale," NBER Working Papers 27298, National Bureau of Economic Research, Inc.
    9. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers: The Impact of Student Access to Wikipedia," CAGE Online Working Paper Series 450, Competitive Advantage in the Global Economy (CAGE).
    10. Chicoine, Luke, 2019. "Schooling with learning: The effect of free primary education and mother tongue instruction reforms in Ethiopia," Economics of Education Review, Elsevier, vol. 69(C), pages 94-107.
    11. Maruyama, Takao & Kurosaki, Takashi, 2021. "Do remedial activities using math workbooks improve student learning? Empirical evidence from scaled-up interventions in Niger," World Development, Elsevier, vol. 148(C).
    12. Adelaide Baronchelli & Alessandra Foresta & Roberto Ricciuti, 2020. "The Words That Keep People Apart. Official Language, Accountability and Fiscal Capacity," CESifo Working Paper Series 8437, CESifo.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Annie Duflo & Jessica Kiessel & Adrienne Lucas, 2020. "Experimental Evidence on Alternative Policies to Increase Learning at Scale," NBER Working Papers 27298, National Bureau of Economic Research, Inc.
    2. Eble, Alex & Frost, Chris & Camara, Alpha & Bouy, Baboucarr & Bah, Momodou & Sivaraman, Maitri & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Brady, Tony & Gawron, Piotr & Vansteelandt, Stijn & Boone, P, 2021. "How much can we remedy very low learning levels in rural parts of low-income countries? Impact and generalizability of a multi-pronged para-teacher intervention from a cluster-randomized trial in the ," Journal of Development Economics, Elsevier, vol. 148(C).
    3. Blimpo, Moussa P. & Pugatch, Todd, 2021. "Entrepreneurship education and teacher training in Rwanda," Journal of Development Economics, Elsevier, vol. 149(C).
    4. Fazzio, Ila & Eble, Alex & Lumsdaine, Robin L. & Boone, Peter & Bouy, Baboucarr & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Johnson, Simon & Silva, Ana Filipa, 2021. "Large learning gains in pockets of extreme poverty: Experimental evidence from Guinea Bissau," Journal of Public Economics, Elsevier, vol. 199(C).
    5. Mo, Di & Bai, Yu & Shi, Yaojiang & Abbey, Cody & Zhang, Linxiu & Rozelle, Scott & Loyalka, Prashant, 2020. "Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China," Journal of Development Economics, Elsevier, vol. 146(C).
    6. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," World Bank Economic Review, World Bank Group, vol. 31(3), pages 687-707.
    7. Sabrin A. Beg & Adrienne M. Lucas & Waqas Halim & Umar Saif, 2019. "Engaging Teachers with Technology Increased Achievement, Bypassing Teachers Did Not," NBER Working Papers 25704, National Bureau of Economic Research, Inc.
    8. Andrew Dustan & Stanislao Maldonado & Juan Manuel Hernandez-Agramonte, 2018. "Motivating bureaucrats with non-monetary incentives when state capacity is weak: Evidence from large-scale field experiments in Peru," Working Papers 136, Peruvian Economic Association.
    9. Cristina Corduneanu-Huci & Michael T. Dorsch & Paul Maarek, 2017. "Learning to constrain: Political competition and randomized controlled trials in development," THEMA Working Papers 2017-24, THEMA (THéorie Economique, Modélisation et Applications), Université de Cergy-Pontoise.
    10. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," World Bank Research Observer, World Bank Group, vol. 33(1), pages 34-64.
    11. Gonzales Mariella & Gianmarco León-Ciliotta & Luis R. Martinez, 2018. "How effective are monetary incentives to vote? Evidence from a nationwide policy," Economics Working Papers 1667, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2019.
    12. Maruyama, Takao & Kurosaki, Takashi, 2021. "Do remedial activities using math workbooks improve student learning? Empirical evidence from scaled-up interventions in Niger," World Development, Elsevier, vol. 148(C).
    13. Afridi, Farzana & Barooah, Bidisha & Somanathan, Rohini, 2020. "Improving learning outcomes through information provision: Experimental evidence from Indian villages," Journal of Development Economics, Elsevier, vol. 146(C).
    14. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    15. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    16. Karthik Muralidharan & Mauricio Romero & Kaspar Wüthrich, 2019. "Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments," NBER Working Papers 26562, National Bureau of Economic Research, Inc.
    17. Avitabile, Ciro & de Hoyos, Rafael, 2018. "The heterogeneous effect of information on student performance: Evidence from a randomized control trial in Mexico," Journal of Development Economics, Elsevier, vol. 135(C), pages 318-348.
    18. Jules Gazeaud & Claire Ricard, 2021. "Conditional cash transfers and the learning crisis: evidence from Tayssir scale-up in Morocco," NOVAFRICA Working Paper Series wp2102, Universidade Nova de Lisboa, Nova School of Business and Economics, NOVAFRICA.
    19. Miguel Urquiola, 2015. "Progress and challenges in achieving an evidence-based education policy in Latin America and the Caribbean," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-30, December.
    20. James Allen IV & Arlete Mahumane & James Riddell IV & Tanya Rosenblat & Dean Yang & Hang Yu, 2021. "Teaching and Incentives: Substitutes or Complements?," NBER Working Papers 28976, National Bureau of Economic Research, Inc.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tpr:restat:v:103:y:2021:i:2:p:251-264. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://direct.mit.edu/journals .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ann Olson (email available below). General contact details of provider: https://direct.mit.edu/journals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.