IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/4036.html
   My bibliography  Save this paper

Evaluating the impact of Mexico's quality schools program : the pitfalls of using nonexperimental data

Author

Listed:
  • Skoufias, Emmanuel
  • Shapiro, Joseph

Abstract

The authors evaluate whether increasing school resources and decentralizing management decisions at the school level improves learning in a developing country. Mexico's Quality Schools Program (PEC), following many other countries and U.S. states, offers US$15,000 grants for public schools to implement five-year improvement plans that the school's staff and community design. Using a three-year panel of 74,700 schools, the authors estimate the impact of the PEC on dropout, repetition, and failure using two common nonexperimental methods-regression analysis and propensity score matching. The methods provide similar but nonidentical results. The preferred estimator, difference-in-differences with matching, reveals that participation in the PEC decreases dropout by 0.24 percentage points, failure by 0.24 percentage points, and repetition by 0.31 percentage points-an economically small but statistically significant impact. The PEC lacks measurable impact on outcomes in indigenous schools. The results suggest that a combination of increased resources and local management can produce small improvements in school outcomes, though perhaps not in the most troubled school systems.

Suggested Citation

  • Skoufias, Emmanuel & Shapiro, Joseph, 2006. "Evaluating the impact of Mexico's quality schools program : the pitfalls of using nonexperimental data," Policy Research Working Paper Series 4036, The World Bank.
  • Handle: RePEc:wbk:wbrwps:4036
    as

    Download full text from publisher

    File URL: http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2006/10/12/000016406_20061012150223/Rendered/PDF/wps4036.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kevin M. Murphy & Finis Welch, 1992. "The Structure of Wages," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 107(1), pages 285-326.
    2. Timothy Besley & Stephen Coate, 1999. "Centralized versus Decentralized Provision of Local Public Goods: A Political Economy Analysis," NBER Working Papers 7084, National Bureau of Economic Research, Inc.
    3. Anne Case & Angus Deaton, 1999. "School Inputs and Educational Outcomes in South Africa," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(3), pages 1047-1084.
    4. Kenneth Y. Chay & Patrick J. McEwan & Miguel Urquiola, 2005. "The Central Role of Noise in Evaluating Interventions That Use Test Scores to Rank Schools," American Economic Review, American Economic Association, vol. 95(4), pages 1237-1258, September.
    5. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    6. Murnane, Richard J. & Willett, John B. & Braatz, M. Jay & Duhaldeborde, Yves, 2001. "Do different dimensions of male high school students' skills predict labor market success a decade later? Evidence from the NLSY," Economics of Education Review, Elsevier, vol. 20(4), pages 311-320, August.
    7. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    8. Alan B. Krueger, 2003. "Economic Considerations and Class Size," Economic Journal, Royal Economic Society, vol. 113(485), pages 34-63, February.
    9. Richard Blundell & Monica Costa Dias, 2000. "Evaluation methods for non-experimental data," Fiscal Studies, Institute for Fiscal Studies, vol. 21(4), pages 427-468, January.
    10. Glewwe, Paul & Kremer, Michael, 2006. "Schools, Teachers, and Education Outcomes in Developing Countries," Handbook of the Economics of Education, in: Erik Hanushek & F. Welch (ed.), Handbook of the Economics of Education, edition 1, volume 2, chapter 16, pages 945-1017, Elsevier.
    11. Jimenez, Emmanuel & Sawada, Yasuyuki, 1999. "Do Community-Managed Schools Work? An Evaluation of El Salvador's EDUCO Program," The World Bank Economic Review, World Bank, vol. 13(3), pages 415-441, September.
    12. O'Neill, June, 1990. "The Role of Human Capital in Earnings Differences between Black and White Men," Journal of Economic Perspectives, American Economic Association, vol. 4(4), pages 25-45, Fall.
    13. Alan B. Krueger, 1999. "Experimental Estimates of Education Production Functions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 497-532.
    14. Alderman, Harold, 2002. "Do local officials know something we don't? Decentralization of targeted transfers in Albania," Journal of Public Economics, Elsevier, vol. 83(3), pages 375-404, March.
    15. Besley, Timothy & Coate, Stephen, 2003. "Centralized versus decentralized provision of local public goods: a political economy approach," Journal of Public Economics, Elsevier, vol. 87(12), pages 2611-2637, December.
    16. James J. Heckman & Jeffrey A. Smith, 1999. "The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies," NBER Working Papers 6983, National Bureau of Economic Research, Inc.
    17. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    18. Murnane, Richard J & Willett, John B & Levy, Frank, 1995. "The Growing Importance of Cognitive Skills in Wage Determination," The Review of Economics and Statistics, MIT Press, vol. 77(2), pages 251-266, May.
    19. Robert J. Barro, 2001. "Human Capital and Growth," American Economic Review, American Economic Association, vol. 91(2), pages 12-17, May.
    20. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 533-575.
    21. Alberto Abadie & Guido W. Imbens, 2008. "On the Failure of the Bootstrap for Matching Estimators," Econometrica, Econometric Society, vol. 76(6), pages 1537-1557, November.
    22. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    23. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    24. Angrist, Joshua D. & Krueger, Alan B., 1999. "Empirical strategies in labor economics," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 23, pages 1277-1366, Elsevier.
    25. Eskeland,Gunnar S. & Filmer,Deon P. & Eskeland, Gunnar S.*Filmer, Deon, 2002. "Autonomy, participation, and learning in Argentine schools - findings and their implications for decentralization," Policy Research Working Paper Series 2766, The World Bank.
    26. Neal, Derek A & Johnson, William R, 1996. "The Role of Premarket Factors in Black-White Wage Differences," Journal of Political Economy, University of Chicago Press, vol. 104(5), pages 869-895, October.
    27. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    28. Jerik Hanushek & Dennis Kimko, 2006. "Schooling, Labor-force Quality, and the Growth of Nations," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 154-193.
    29. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cabrera Hernández, Francisco-Javier, 2016. "Essays on the impact evaluation of education policies in Mexico," Economics PhD Theses 0316, Department of Economics, University of Sussex Business School.
    2. Bando, Rosangela, 2015. "The effect of cash transfers to schools on voluntary contributions," Journal of Economic Behavior & Organization, Elsevier, vol. 111(C), pages 224-236.
    3. Francisco Cabrera-Hernandez, 2015. "Does lengthening the school day increase students’ academic achievement? Evidence from a natural experiment," Working Paper Series 7415, Department of Economics, University of Sussex Business School.
    4. Al-Samarrai, Samer & Shrestha, Unika & Hasan, Amer & Nakajima, Nozomi & Santoso, Santoso & Wijoyo, Wisnu Harto Adi, 2018. "Introducing a performance-based component into Jakarta's school grants: What do we know about its impact after three years?," Economics of Education Review, Elsevier, vol. 67(C), pages 110-136.
    5. Panchali Guha, 2023. "School committee composition: Exploring the role of parental and female representation in India," Development Policy Review, Overseas Development Institute, vol. 41(3), May.
    6. Bui, Tuan Anh & Nguyen, Cuong Viet & Nguyen, Khuong Duc & Nguyen, Ha Hong & Pham, Phuong Thu, 2020. "The effect of tuition fee reduction and education subsidy on school enrollment: Evidence from Vietnam," Children and Youth Services Review, Elsevier, vol. 108(C).
    7. World Bank Group, 2014. "Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations," World Bank Publications - Books, The World Bank Group, number 21113, December.
    8. Padilla-Romo, María, 2022. "Full-time schools, policy-induced school switching, and academic performance," Journal of Economic Behavior & Organization, Elsevier, vol. 196(C), pages 79-103.
    9. Bando, Rosangela, 2015. "The effect of cash transfers to schools on voluntary contributions," Journal of Economic Behavior & Organization, Elsevier, vol. 111(C), pages 224-236.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    3. Eric A. Hanushek, 2008. "Incentives for Efficiency and Equity in the School System," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(s1), pages 5-27, May.
    4. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    5. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    6. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    7. Michael Lechner, 2000. "An Evaluation of Public-Sector-Sponsored Continuous Vocational Training Programs in East Germany," Journal of Human Resources, University of Wisconsin Press, vol. 35(2), pages 347-375.
    8. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    9. Jochen Kluve & Boris Augurzky, 2007. "Assessing the performance of matching algorithms when selection into treatment is strong," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(3), pages 533-557.
    10. Eric A. Hanushek & Victor Lavy & Kohtaro Hitomi, 2008. "Do Students Care about School Quality? Determinants of Dropout Behavior in Developing Countries," Journal of Human Capital, University of Chicago Press, vol. 2(1), pages 69-105.
    11. Paula Salinas & Albert Solé-Ollé, 2009. "Evaluating the effects of decentralization on educational outcomes in Spain," Working Papers 2009/10, Institut d'Economia de Barcelona (IEB).
    12. Glewwe, Paul & Kremer, Michael, 2006. "Schools, Teachers, and Education Outcomes in Developing Countries," Handbook of the Economics of Education, in: Erik Hanushek & F. Welch (ed.), Handbook of the Economics of Education, edition 1, volume 2, chapter 16, pages 945-1017, Elsevier.
    13. Eric A. Hanushek & Ludger Woessmann, 2008. "The Role of Cognitive Skills in Economic Development," Journal of Economic Literature, American Economic Association, vol. 46(3), pages 607-668, September.
    14. Bryson, Alex & Dorsett, Richard & Purdon, Susan, 2002. "The use of propensity score matching in the evaluation of active labour market policies," LSE Research Online Documents on Economics 4993, London School of Economics and Political Science, LSE Library.
    15. Advani, Arun & Sloczynski, Tymon, 2013. "Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies," IZA Discussion Papers 7874, Institute of Labor Economics (IZA).
    16. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
    17. Behrman, Jere R., 2010. "Investment in Education Inputs and Incentives," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4883-4975, Elsevier.
    18. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.
    19. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    20. Justine Burns & Malcolm Kewsell & Rebecca Thornton, 2009. "Evaluating the Impact of Health Programmes," SALDRU Working Papers 40, Southern Africa Labour and Development Research Unit, University of Cape Town.

    More about this item

    Keywords

    Tertiary Education; Education For All; Primary Education; Teaching and Learning; Secondary Education; Economics of Education;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:4036. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Roula I. Yazigi (email available below). General contact details of provider: https://edirc.repec.org/data/dvewbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.