IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v290y2021i3p1111-1124.html
   My bibliography  Save this article

Impact evaluation in a multi-input multi-output setting: Evidence on the effect of additional resources for schools

Author

Listed:
  • D’Inverno, Giovanna
  • Smet, Mike
  • De Witte, Kristof

Abstract

This paper proposes an innovative approach to evaluate the causal impact of a policy change in a multi-input multi-output setting. It combines insights from econometric impact evaluation techniques and efficiency analysis. In particular, the current paper accounts for endogeneity issues by introducing a quasi-experimental setting within a conditional multi-input multi-output efficiency framework and by decomposing the overall efficiency between ‘group-specific’ efficiency (i.e., reflecting internal managerial inefficiency) and ‘program’ efficiency (i.e., explaining the impact of the policy intervention on performance). This framework allows the researcher to interpret the efficiency scores in terms of causality. The practical usefulness of the methodology is demonstrated through an application to secondary schools in Flanders, Belgium. By exploiting an exogenous threshold, the paper examines whether additional resources for disadvantaged students impact the efficiency of schools. The empirical results indicate that additional resources do not causally influence efficiency around the threshold.

Suggested Citation

  • D’Inverno, Giovanna & Smet, Mike & De Witte, Kristof, 2021. "Impact evaluation in a multi-input multi-output setting: Evidence on the effect of additional resources for schools," European Journal of Operational Research, Elsevier, vol. 290(3), pages 1111-1124.
  • Handle: RePEc:eee:ejores:v:290:y:2021:i:3:p:1111-1124
    DOI: 10.1016/j.ejor.2020.08.042
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221720307554
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ejor.2020.08.042?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. D'Inverno, Giovanna & Carosi, Laura & Ravagli, Letizia, 2018. "Global public spending efficiency in Tuscan municipalities," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 102-113.
    2. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    3. Gordon B. Dahl & Lance Lochner, 2012. "The Impact of Family Income on Child Achievement: Evidence from the Earned Income Tax Credit," American Economic Review, American Economic Association, vol. 102(5), pages 1927-1956, August.
    4. Simar, Léopold & Vanhems, Anne & Van Keilegom, Ingrid, 2016. "Unobserved heterogeneity and endogeneity in nonparametric frontier estimation," Journal of Econometrics, Elsevier, vol. 190(2), pages 360-373.
    5. Jörn-Steffen Pischke & Till von Wachter, 2008. "Zero Returns to Compulsory Schooling in Germany: Evidence and Interpretation," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 592-598, August.
    6. Kerstens, Kristiaan & O’Donnell, Christopher & Van de Woestyne, Ignace, 2019. "Metatechnology frontier and convexity: A restatement," European Journal of Operational Research, Elsevier, vol. 275(2), pages 780-792.
    7. C. J. O’Donnell & Saeideh Fallah-Fini & Konstantinos Triantis, 2017. "Measuring and analysing productivity change in a metafrontier framework," Journal of Productivity Analysis, Springer, vol. 47(2), pages 117-128, April.
    8. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Rocío Titiunik, 2019. "Regression Discontinuity Designs Using Covariates," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 442-451, July.
    9. Markus Frölich & Martin Huber, 2019. "Including Covariates in the Regression Discontinuity Design," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(4), pages 736-748, October.
    10. Bădin, Luiza & Daraio, Cinzia & Simar, Léopold, 2012. "How to measure the impact of environmental factors in a nonparametric production model," European Journal of Operational Research, Elsevier, vol. 223(3), pages 818-833.
    11. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    12. Henry M. Levin, 1974. "Measuring Efficiency in Educational Production," Public Finance Review, , vol. 2(1), pages 3-24, January.
    13. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust data-driven inference in the regression-discontinuity design," Stata Journal, StataCorp LP, vol. 14(4), pages 909-946, December.
    14. Julien Grenet, 2013. "Is Extending Compulsory Schooling Alone Enough to Raise Earnings? Evidence from French and British Compulsory Schooling Laws," Scandinavian Journal of Economics, Wiley Blackwell, vol. 115(1), pages 176-210, January.
    15. Piero Cipollone & Alfonso Rosolia, 2007. "Social Interactions in High School: Lessons from an Earthquake," American Economic Review, American Economic Association, vol. 97(3), pages 948-965, June.
    16. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    17. Daniel Santín & Gabriela Sicilia, 2018. "Using DEA for measuring teachers’ performance and the impact on students’ outcomes: evidence for Spain," Journal of Productivity Analysis, Springer, vol. 49(1), pages 1-15, February.
    18. Afriat, Sidney N, 1972. "Efficiency Estimation of Production Function," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 13(3), pages 568-598, October.
    19. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    20. Daneshvary, Nasser & Clauretie, Terrence M., 2001. "Efficiency and costs in education: year-round versus traditional schedules," Economics of Education Review, Elsevier, vol. 20(3), pages 279-287, June.
    21. Melvin Stephens Jr. & Dou-Yan Yang, 2014. "Compulsory Education and the Benefits of Schooling," American Economic Review, American Economic Association, vol. 104(6), pages 1777-1792, June.
    22. Duflo, Esther & Dupas, Pascaline & Kremer, Michael, 2015. "School governance, teacher incentives, and pupil–teacher ratios: Experimental evidence from Kenyan primary schools," Journal of Public Economics, Elsevier, vol. 123(C), pages 92-110.
    23. Alois Kneip & Léopold Simar & Paul W. Wilson, 2016. "Testing Hypotheses in Nonparametric Models of Production," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(3), pages 435-456, July.
    24. C B Vaz & A S Camanho, 2012. "Performance comparison of retailing stores using a Malmquist-type index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 63(5), pages 631-645, May.
    25. Eric A. Hanushek, 1979. "Conceptual and Empirical Issues in the Estimation of Educational Production Functions," Journal of Human Resources, University of Wisconsin Press, vol. 14(3), pages 351-388.
    26. Amsler, Christine & Prokhorov, Artem & Schmidt, Peter, 2016. "Endogeneity in stochastic frontier models," Journal of Econometrics, Elsevier, vol. 190(2), pages 280-288.
    27. Hanushek, Eric A., 2002. "Publicly provided education," Handbook of Public Economics, in: A. J. Auerbach & M. Feldstein (ed.), Handbook of Public Economics, edition 1, volume 4, chapter 30, pages 2045-2141, Elsevier.
    28. Cinzia Daraio & Léopold Simar, 2005. "Introducing Environmental Variables in Nonparametric Frontier Models: a Probabilistic Approach," Journal of Productivity Analysis, Springer, vol. 24(1), pages 93-121, September.
    29. Edwin Leuven & Mikael Lindahl & Hessel Oosterbeek & Dinand Webbink, 2007. "The Effect of Extra Funding for Disadvantaged Pupils on Achievement," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 721-736, November.
    30. Stephen Gibbons & Sandra McNally & Martina Viarengo, 2018. "Does Additional Spending Help Urban Schools? An Evaluation Using Boundary Discontinuities," Journal of the European Economic Association, European Economic Association, vol. 16(5), pages 1618-1668.
    31. Olesen, Ole Bent & Petersen, Niels Christian & Podinovski, Victor V., 2017. "Efficiency measures and computational approaches for data envelopment analysis models with ratio inputs and outputs," European Journal of Operational Research, Elsevier, vol. 261(2), pages 640-655.
    32. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    33. Mark Doms & Eric J. Bartelsman, 2000. "Understanding Productivity: Lessons from Longitudinal Microdata," Journal of Economic Literature, American Economic Association, vol. 38(3), pages 569-594, September.
    34. Cordero, José Manuel & Santín, Daniel & Sicilia, Gabriela, 2015. "Testing the accuracy of DEA estimates under endogeneity through a Monte Carlo simulation," European Journal of Operational Research, Elsevier, vol. 244(2), pages 511-518.
    35. Michael Rosenfeld, 2010. "Nontraditional families and childhood progress through school," Demography, Springer;Population Association of America (PAA), vol. 47(3), pages 755-775, August.
    36. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 933-959.
    37. Martin Schlotter & Guido Schwerdt & Ludger Woessmann, 2011. "Econometric methods for causal evaluation of education policies and practices: a non-technical guide," Education Economics, Taylor & Francis Journals, vol. 19(2), pages 109-137.
    38. Andrew Johnson & John Ruggiero, 2014. "Nonparametric measurement of productivity and efficiency in education," Annals of Operations Research, Springer, vol. 221(1), pages 197-210, October.
    39. Kneip, Alois & Simar, Léopold & Wilson, Paul W., 2015. "When Bias Kills The Variance: Central Limit Theorems For Dea And Fdh Efficiency Scores," Econometric Theory, Cambridge University Press, vol. 31(2), pages 394-422, April.
    40. A. Camanho & R. Dyson, 2006. "Data envelopment analysis and Malmquist indices for measuring group performance," Journal of Productivity Analysis, Springer, vol. 26(1), pages 35-49, August.
    41. Aparicio, Juan & Crespo-Cebada, Eva & Pedraja-Chaparro, Francisco & Santín, Daniel, 2017. "Comparing school ownership performance using a pseudo-panel database: A Malmquist-type index approach," European Journal of Operational Research, Elsevier, vol. 256(2), pages 533-542.
    42. Grosskopf, Shawna & Moutray, Chad, 2001. "Evaluating performance in Chicago public high schools in the wake of decentralization," Economics of Education Review, Elsevier, vol. 20(1), pages 1-14, February.
    43. Christopher O’Donnell & D. Rao & George Battese, 2008. "Metafrontier frameworks for the study of firm-level efficiencies and technology ratios," Empirical Economics, Springer, vol. 34(2), pages 231-255, March.
    44. Cinzia Daraio & Léopold Simar, 2007. "Conditional nonparametric frontier models for convex and nonconvex technologies: a unifying approach," Journal of Productivity Analysis, Springer, vol. 28(1), pages 13-32, October.
    45. De Witte, Kristof & Rogge, Nicky, 2011. "Accounting for exogenous influences in performance evaluations of teachers," Economics of Education Review, Elsevier, vol. 30(4), pages 641-653, August.
    46. Geraint Johnes & Jill Johnes & Tommaso Agasisti & Laura López-Torres (ed.), 2017. "Handbook of Contemporary Education Economics," Books, Edward Elgar Publishing, number 17209.
    47. Jill Johnes & Maria Portela & Emmanuel Thanassoulis, 2017. "Efficiency in education," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 331-338, April.
    48. Aparicio, Juan & Santin, Daniel, 2018. "A note on measuring group performance over time with pseudo-panels," European Journal of Operational Research, Elsevier, vol. 267(1), pages 227-235.
    49. A. Charnes & W. W. Cooper & E. Rhodes, 1981. "Evaluating Program and Managerial Efficiency: An Application of Data Envelopment Analysis to Program Follow Through," Management Science, INFORMS, vol. 27(6), pages 668-697, June.
    50. Kristof De Witte & Mika Kortelainen, 2013. "What explains the performance of students in a heterogeneous environment? Conditional efficiency estimation with continuous and discrete environmental variables," Applied Economics, Taylor & Francis Journals, vol. 45(17), pages 2401-2412, June.
    51. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Roc ́ıo Titiunik, 2017. "rdrobust: Software for regression-discontinuity designs," Stata Journal, StataCorp LP, vol. 17(2), pages 372-404, June.
    52. George Battese & D. Rao & Christopher O'Donnell, 2004. "A Metafrontier Production Function for Estimation of Technical Efficiencies and Technology Gaps for Firms Operating Under Different Technologies," Journal of Productivity Analysis, Springer, vol. 21(1), pages 91-103, January.
    53. McCrary, Justin, 2008. "Manipulation of the running variable in the regression discontinuity design: A density test," Journal of Econometrics, Elsevier, vol. 142(2), pages 698-714, February.
    54. Cazals, Catherine & Fève, Frédérique & Florens, Jean-Pierre & Simar, Léopold, 2016. "Nonparametric instrumental variables estimation for efficiency frontier," Journal of Econometrics, Elsevier, vol. 190(2), pages 349-359.
    55. Bradley, Steve & Johnes, Geraint & Millington, Jim, 2001. "The effect of competition on the efficiency of secondary schools in England," European Journal of Operational Research, Elsevier, vol. 135(3), pages 545-568, December.
    56. Daniel Santín & Gabriela Sicilia, 2017. "Impact evaluation and frontier methods in education: a step forward," Chapters, in: Geraint Johnes & Jill Johnes & Tommaso Agasisti & Laura López-Torres (ed.), Handbook of Contemporary Education Economics, chapter 10, pages 211-245, Edward Elgar Publishing.
    57. George E. Battese & D. S. Prasada Rao, 2002. "Technology Gap, Efficiency, and a Stochastic Metafrontier Function," International Journal of Business and Economics, School of Management Development, Feng Chia University, Taichung, Taiwan, vol. 1(2), pages 87-93, August.
    58. C. Kirabo Jackson & Rucker C. Johnson & Claudia Persico, 2016. "The Effects of School Spending on Educational and Economic Outcomes: Evidence from School Finance Reforms," The Quarterly Journal of Economics, Oxford University Press, vol. 131(1), pages 157-218.
    59. Olesen, Ole Bent & Petersen, Niels Christian & Podinovski, Victor V., 2015. "Efficiency analysis with ratio measures," European Journal of Operational Research, Elsevier, vol. 245(2), pages 446-462.
    60. Karthik Muralidharan & Venkatesh Sundararaman, 2015. "Editor's Choice The Aggregate Effect of School Choice: Evidence from a Two-Stage Experiment in India," The Quarterly Journal of Economics, Oxford University Press, vol. 130(3), pages 1011-1066.
    61. Laurens Cherchye & Kristof De Witte & Sergio Perelman, 2019. "A unified productivity-performance approach applied to secondary schools," Journal of the Operational Research Society, Taylor & Francis Journals, vol. 70(9), pages 1522-1537, September.
    62. D J Mayston, 2003. "Measuring and managing educational performance," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 54(7), pages 679-691, July.
    63. Cazals, Catherine & Florens, Jean-Pierre & Simar, Leopold, 2002. "Nonparametric frontier estimation: a robust approach," Journal of Econometrics, Elsevier, vol. 106(1), pages 1-25, January.
    64. Grosskopf, S. & Valdmanis, V., 1987. "Measuring hospital performance : A non-parametric approach," Journal of Health Economics, Elsevier, vol. 6(2), pages 89-107, June.
    65. Shawna Grosskopf & Kathy J. Hayes & Lori L. Taylor, 2014. "Efficiency in Education: Research and Implications," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(2), pages 175-210.
    66. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 114(2), pages 533-575.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    2. José Manuel Cordero & Daniel Santín & Rosa Simancas, 2017. "Assessing European primary school performance through a conditional nonparametric model," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 364-376, April.
    3. Aparicio, Juan & Ortiz, Lidia & Santín, Daniel, 2021. "Comparing group performance over time through the Luenberger productivity indicator: An application to school ownership in European countries," European Journal of Operational Research, Elsevier, vol. 294(2), pages 651-672.
    4. Cordero, Jose M. & Polo, Cristina & Santín, Daniel & Simancas, Rosa, 2018. "Efficiency measurement and cross-country differences among schools: A robust conditional nonparametric analysis," Economic Modelling, Elsevier, vol. 74(C), pages 45-60.
    5. López-Torres, Laura & Johnes, Jill & Elliott, Caroline & Polo, Cristina, 2021. "The effects of competition and collaboration on efficiency in the UK independent school sector," Economic Modelling, Elsevier, vol. 96(C), pages 40-53.
    6. José Manuel Cordero & Diego Prior & Rosa Simancas, 2016. "A comparison of public and private schools in Spain using robust nonparametric frontier methods," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 24(3), pages 659-680, September.
    7. Daraio, Cinzia & Simar, Léopold & Wilson, Paul W., 2021. "Quality as a latent heterogeneity factor in the efficiency of universities," Economic Modelling, Elsevier, vol. 99(C).
    8. Blaise Melly & Rafael Lalive, 2020. "Estimation, Inference, and Interpretation in the Regression Discontinuity Design," Diskussionsschriften dp2016, Universitaet Bern, Departement Volkswirtschaft.
    9. Fritz Schiltz & Kristof Witte & Deni Mazrekaj, 2020. "Managerial efficiency and efficiency differentials in adult education: a conditional and bias-corrected efficiency analysis," Annals of Operations Research, Springer, vol. 288(2), pages 529-546, May.
    10. Aparicio, Juan & Crespo-Cebada, Eva & Pedraja-Chaparro, Francisco & Santín, Daniel, 2017. "Comparing school ownership performance using a pseudo-panel database: A Malmquist-type index approach," European Journal of Operational Research, Elsevier, vol. 256(2), pages 533-542.
    11. Kerstens, Kristiaan & O’Donnell, Christopher & Van de Woestyne, Ignace, 2019. "Metatechnology frontier and convexity: A restatement," European Journal of Operational Research, Elsevier, vol. 275(2), pages 780-792.
    12. López-Torres, Laura & Nicolini, Rosella & Prior, Diego, 2017. "Does strategic interaction affect demand for school places? A conditional efficiency approach," Regional Science and Urban Economics, Elsevier, vol. 65(C), pages 89-103.
    13. Angelo D'Andrea, 2019. "Mayor’s wage and Public procurement," BAFFI CAREFIN Working Papers 19125, BAFFI CAREFIN, Centre for Applied Research on International Markets Banking Finance and Regulation, Universita' Bocconi, Milano, Italy.
    14. Mellace, Giovanni & Ventura, Marco, 2019. "Intended and unintended effects of public incentives for innovation. Quasi-experimental evidence from Italy," Discussion Papers of Business and Economics 9/2019, University of Southern Denmark, Department of Business and Economics.
    15. Afsharian, Mohsen & Ahn, Heinz & Harms, Sören Guntram, 2019. "Performance comparison of management groups under centralised management," European Journal of Operational Research, Elsevier, vol. 278(3), pages 845-854.
    16. Meyer, Andrew G., 2017. "The impact of education on political ideology: Evidence from European compulsory education reforms," Economics of Education Review, Elsevier, vol. 56(C), pages 9-23.
    17. Valentin Zelenyuk, 2019. "Data Envelopment Analysis and Business Analytics: The Big Data Challenges and Some Solutions," CEPA Working Papers Series WP072019, School of Economics, University of Queensland, Australia.
    18. Paola Azar & Gabriela Sicilia, 2021. "An assessment of pupil and school performance in public primary education in Uruguay," Documentos de Trabajo (working papers) 21-22, Instituto de Economia - IECON.
    19. De Witte, Kristof & Schiltz, Fritz, 2018. "Measuring and explaining organizational effectiveness of school districts: Evidence from a robust and conditional Benefit-of-the-Doubt approach," European Journal of Operational Research, Elsevier, vol. 267(3), pages 1172-1181.
    20. Bahadır Dursun & Resul Cesur, 2016. "Transforming lives: the impact of compulsory schooling on hope and happiness," Journal of Population Economics, Springer;European Society for Population Economics, vol. 29(3), pages 911-956, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:290:y:2021:i:3:p:1111-1124. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.elsevier.com/locate/eor .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.