IDEAS home Printed from https://ideas.repec.org/p/fbk/wpaper/2014-05.html
   My bibliography  Save this paper

Counting Rotten Apples: Student Achievement and Score Manipulation in Italian Elementary Schools

Author

Listed:
  • Erich Battistin
  • Michele De Nadai
  • Daniela Vuri

Abstract

We derive bounds for the average of math and language scores of elementary school students in Italy correcting for pervasive score manipulation. Information on the fraction of manipulated data is retrieved from a natural experiment that randomly assigns external monitors to schools. We show how bounds can be tightened imposing restrictions on the measurement properties of the manipulation indicator developed by the government agency charged with test administration and data collection. We additionally assume that manipulation is more likely in those classes at the lower end of the distribution of true scores. Our results show that regional rankings by academic performance are reversed once manipulation is properly taken into account.

Suggested Citation

  • Erich Battistin & Michele De Nadai & Daniela Vuri, 2014. "Counting Rotten Apples: Student Achievement and Score Manipulation in Italian Elementary Schools," FBK-IRVAPP Working Papers 2014-05, Research Institute for the Evaluation of Public Policies (IRVAPP), Bruno Kessler Foundation.
    • Handle: RePEc:fbk:wpaper:2014-05
    as

    Download full text from publisher

    File URL: https://irvapp.fbk.eu/wp-content/uploads/2017/10/wp_irvapp_2014-05.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anders Böhlmark & Mikael Lindahl, 2015. "Independent Schools and Long-run Educational Outcomes: Evidence from Sweden's Large-scale Voucher Reform," Economica, London School of Economics and Political Science, vol. 82(327), pages 508-551, July.
    2. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    3. Andrea Ichino & Giovanni Maggi, 2000. "Work Environment and Individual Background: Explaining Regional Shirking Differentials in a Large Italian Firm," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(3), pages 1057-1090.
    4. Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), 2011. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 4, number 4, June.
    5. Aigner, Dennis J., 1973. "Regression with a binary independent variable subject to errors of observation," Journal of Econometrics, Elsevier, vol. 1(1), pages 49-59, March.
    6. Joshua D. Angrist & Parag A. Pathak & Christopher R. Walters, 2013. "Explaining Charter School Effectiveness," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 1-27, October.
    7. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2014. "In a Small Moment: Class Size and Moral Hazard in the Mezzogiorno," FBK-IRVAPP Working Papers 2014-04, Research Institute for the Evaluation of Public Policies (IRVAPP), Bruno Kessler Foundation.
    8. Hanushek, Eric A. & Woessmann, Ludger, 2011. "The Economics of International Differences in Educational Achievement," Handbook of the Economics of Education, in: Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), Handbook of the Economics of Education, edition 1, volume 3, chapter 2, pages 89-200, Elsevier.
    9. Kreider, Brent & Pepper, John V., 2011. "Identification of Expected Outcomes in a Data Error Mixing Model With Multiplicative Mean Independence," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(1), pages 49-60.
    10. Rebecca Diamond & Petra Persson, 2016. "The Long-term Consequences of Teacher Discretion in Grading of High-stakes Tests," NBER Working Papers 22207, National Bureau of Economic Research, Inc.
    11. Aviv Nevo & Adam M. Rosen, 2012. "Identification With Imperfect Instruments," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 659-671, August.
    12. Kreider, Brent & Pepper, John V., 2007. "Disability and Employment: Reevaluating the Evidence in Light of Reporting Errors," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 432-441, June.
    13. Brian A. Jacob & Steven D. Levitt, 2003. "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 843-877.
    14. Guido W. Imbens & Charles F. Manski, 2004. "Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 72(6), pages 1845-1857, November.
    15. Nicoletti, Cheti & Peracchi, Franco & Foliano, Francesca, 2011. "Estimating Income Poverty in the Presence of Missing Data and Measurement Error," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(1), pages 61-72.
    16. Battistin, Erich & De Nadai, Michele & Sianesi, Barbara, 2014. "Misreported schooling, multiple measures and returns to educational qualifications," Journal of Econometrics, Elsevier, vol. 181(2), pages 136-150.
    17. Battistin, Erich & Meroni, Elena Claudia, 2016. "Should we increase instruction time in low achieving schools? Evidence from Southern Italy," Economics of Education Review, Elsevier, vol. 55(C), pages 39-56.
    18. Arthur Lewbel, 2007. "Estimation of Average Treatment Effects with Misclassification," Econometrica, Econometric Society, vol. 75(2), pages 537-551, March.
    19. Horowitz, Joel L & Manski, Charles F, 1995. "Identification and Robustness with Contaminated and Corrupted Data," Econometrica, Econometric Society, vol. 63(2), pages 281-302, March.
    20. Thomas J. Kane & Cecilia Elena Rouse & Douglas Staiger, 1999. "Estimating Returns to Schooling When Schooling is Misreported," NBER Working Papers 7235, National Bureau of Economic Research, Inc.
    21. Bratti, Massimiliano & Checchi, Daniele & Filippin, Antonio, 2007. "Territorial Differences in Italian Students’ Mathematical Competencies: Evidence from PISA 2003," IZA Discussion Papers 2603, Institute of Labor Economics (IZA).
    22. Frolich, Markus, 2007. "Nonparametric IV estimation of local average treatment effects with covariates," Journal of Econometrics, Elsevier, vol. 139(1), pages 35-75, July.
    23. Hahn, Jinyong, 1996. "A Note on Bootstrapping Generalized Method of Moments Estimators," Econometric Theory, Cambridge University Press, vol. 12(1), pages 187-197, March.
    24. Luigi Guiso & Paola Sapienza & Luigi Zingales, 2004. "The Role of Social Capital in Financial Development," American Economic Review, American Economic Association, vol. 94(3), pages 526-556, June.
    25. Brent Kreider & John V. Pepper & Craig Gundersen & Dean Jolliffe, 2012. "Identifying the Effects of SNAP (Food Stamps) on Child Health Outcomes When Participation Is Endogenous and Misreported," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(499), pages 958-975, September.
    26. Tommaso Nannicini & Andrea Stella & Guido Tabellini & Ugo Troiano, 2013. "Social Capital and Political Accountability," American Economic Journal: Economic Policy, American Economic Association, vol. 5(2), pages 222-250, May.
    27. Charles F. Manski & John V. Pepper, 2000. "Monotone Instrumental Variables, with an Application to the Returns to Schooling," Econometrica, Econometric Society, vol. 68(4), pages 997-1012, July.
    28. Hu, Yingyao, 2008. "Identification and estimation of nonlinear models with misclassification error using instrumental variables: A general solution," Journal of Econometrics, Elsevier, vol. 144(1), pages 27-61, May.
    29. Xiaohong Chen & Han Hong & Denis Nekipelov, 2011. "Nonlinear Models of Measurement Errors," Journal of Economic Literature, American Economic Association, vol. 49(4), pages 901-937, December.
    30. Thomas S. Dee & Will Dobbie & Brian A. Jacob & Jonah Rockoff, 2019. "The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations," American Economic Journal: Applied Economics, American Economic Association, vol. 11(3), pages 382-423, July.
    31. Bertoni, Marco & Brunello, Giorgio & Rocco, Lorenzo, 2013. "When the cat is near, the mice won't play: The effect of external examiners in Italian schools," Journal of Public Economics, Elsevier, vol. 104(C), pages 65-77.
    32. Molinari, Francesca, 2008. "Partial identification of probability distributions with misclassified data," Journal of Econometrics, Elsevier, vol. 144(1), pages 81-117, May.
    33. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.
    34. Luigi Cannari & Francesco Nucci & Paolo Sestito, 2000. "Geographic labour mobility and the cost of housing: evidence from Italy," Applied Economics, Taylor & Francis Journals, vol. 32(14), pages 1899-1906.
    35. AIGNER, Dennis J., 1973. "Regression with a binary independent variable subject to errors of observation," LIDAM Reprints CORE 130, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    36. Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), 2011. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 3, number 3, June.
    37. Aprajit Mahajan, 2006. "Identification and Estimation of Regression Models with Misclassification," Econometrica, Econometric Society, vol. 74(3), pages 631-665, May.
    38. Thomas J. Kane & Cecilia Rouse & Douglas Staiger, 1999. "Estimating Returns to Schooling When Schooling is Misreported," Working Papers 798, Princeton University, Department of Economics, Industrial Relations Section..
    39. Abadie A., 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 284-292, March.
    40. Robert P. Sherman & Jeff Dominitz, 2006. "Identification and estimation of bounds on school performance measures: a nonparametric analysis of a mixture model with verification," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(8), pages 1295-1326.
    41. Di Addario, Sabrina & Patacchini, Eleonora, 2008. "Wages and the City. Evidence from Italy," Labour Economics, Elsevier, vol. 15(5), pages 1040-1061, October.
    42. Edward Vytlacil, 2002. "Independence, Monotonicity, and Latent Index Models: An Equivalence Result," Econometrica, Econometric Society, vol. 70(1), pages 331-341, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marina Cavalieri & Massimo Finocchiaro Castro & Calogero Guccio, 2020. "Does the Fish Rot from the Head? Organised Crime and Educational Outcomes in Southern Italy," Working papers 97, Società Italiana di Economia Pubblica.
    2. Veronica Minaya & Tommaso Agasisti, 2019. "Evaluating the Stability of School Performance Estimates over Time," Fiscal Studies, John Wiley & Sons, vol. 40(3), pages 401-425, September.
    3. Martin Gustafsson & Carol Nuga Deliwe, 2017. "Rotten apples or just apples and pears? Understanding patterns consistent with cheating in international test data," Working Papers 17/2017, Stellenbosch University, Department of Economics.
    4. Erich Battistin, 2016. "How manipulating test scores affects school accountability and student achievement," IZA World of Labor, Institute of Labor Economics (IZA), pages 295-295, September.
    5. Bertoni, Marco & Brunello, Giorgio & De Benedetto, Marco Alberto & De Paola, Maria, 2019. "External Monitors and Score Manipulation in Italian Schools: Symptomatic Treatment or Cure?," IZA Discussion Papers 12591, Institute of Labor Economics (IZA).
    6. Carmen Aina & Massimiliano Bratti & Enrico Lippo, 2021. "Ranking high schools using university student performance in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 38(1), pages 293-321, April.
    7. Santiago Pereda Fernández, 2016. "A new method for the correction of test scores manipulation," Temi di discussione (Economic working papers) 1047, Bank of Italy, Economic Research and International Relations Area.
    8. Claudio Lucifora & Marco Tonello, 2020. "Monitoring and Sanctioning Cheating at School: What Works? Evidence from a National Evaluation Program," Journal of Human Capital, University of Chicago Press, vol. 14(4), pages 584-616.
    9. Cavalieri, Marina & Finocchiaro Castro, Massimo & Guccio, Calogero, 2023. "Organised crime and educational outcomes in Southern Italy: An empirical investigation," Socio-Economic Planning Sciences, Elsevier, vol. 89(C).
    10. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.
    2. Battistin, Erich & De Nadai, Michele & Sianesi, Barbara, 2014. "Misreported schooling, multiple measures and returns to educational qualifications," Journal of Econometrics, Elsevier, vol. 181(2), pages 136-150.
    3. Tommasi, Denni & Zhang, Lina, 2024. "Bounding program benefits when participation is misreported," Journal of Econometrics, Elsevier, vol. 238(1).
    4. Denni Tommasi & Arthur Lewbel & Rossella Calvi, 2017. "LATE with Mismeasured or Misspecified Treatment: An application to Women's Empowerment in India," Working Papers ECARES ECARES 2017-27, ULB -- Universite Libre de Bruxelles.
    5. Takahide Yanagi, 2019. "Inference on local average treatment effects for misclassified treatment," Econometric Reviews, Taylor & Francis Journals, vol. 38(8), pages 938-960, September.
    6. Francis DiTraglia & Camilo Garcia-Jimeno, 2015. "On Mis-measured Binary Regressors: New Results And Some Comments on the Literature, Third Version," PIER Working Paper Archive 15-040, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania, revised 24 Nov 2015.
    7. Acerenza, Santiago & Ban, Kyunghoon & Kedagni, Desire, 2021. "Marginal Treatment Effects with Misclassified Treatment," ISU General Staff Papers 202106180700001132, Iowa State University, Department of Economics.
    8. DiTraglia, Francis J. & García-Jimeno, Camilo, 2019. "Identifying the effect of a mis-classified, binary, endogenous regressor," Journal of Econometrics, Elsevier, vol. 209(2), pages 376-390.
    9. Francis J. DiTraglia & Camilo García-Jimeno, 2017. "Mis-classified, Binary, Endogenous Regressors: Identification and Inference," NBER Working Papers 23814, National Bureau of Economic Research, Inc.
    10. Akanksha Negi & Digvijay Singh Negi, 2022. "Difference-in-Differences with a Misclassified Treatment," Papers 2208.02412, arXiv.org.
    11. Orville Mondal & Rui Wang, 2024. "Partial Identification of Binary Choice Models with Misreported Outcomes," Papers 2401.17137, arXiv.org.
    12. Francis J. DiTraglia & Camilo Garcia-Jimeno, 2020. "Identifying the effect of a mis-classified, binary, endogenous regressor," Papers 2011.07272, arXiv.org.
    13. Francis DiTraglia & Camilo Garcia-Jimeno, 2015. "On Mis-measured Binary Regressors: New Results And Some Comments on the Literature, Second Version," PIER Working Paper Archive 15-039, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania, revised 11 Nov 2015.
    14. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2014. "In a Small Moment: Class Size and Moral Hazard in the Mezzogiorno," FBK-IRVAPP Working Papers 2014-04, Research Institute for the Evaluation of Public Policies (IRVAPP), Bruno Kessler Foundation.
    15. Steven J. Haider & Melvin Stephens Jr., 2020. "Correcting for Misclassified Binary Regressors Using Instrumental Variables," NBER Working Papers 27797, National Bureau of Economic Research, Inc.
    16. Wossen, Tesfamicheal & Abay, Kibrom A. & Abdoulaye, Tahirou, 2022. "Misperceiving and misreporting input quality: Implications for input use and productivity," Journal of Development Economics, Elsevier, vol. 157(C).
    17. Erich Battistin & Barbara Sianesi, 2006. "Misreported schooling and returns to education: evidence from the UK," CeMMAP working papers CWP07/06, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    18. Francis J. DiTraglia & Camilo Garcia-Jimeno, 2020. "A Framework for Eliciting, Incorporating, and Disciplining Identification Beliefs in Linear Models," Papers 2011.07276, arXiv.org.
    19. Kasahara, Hiroyuki & Shimotsu, Katsumi, 2022. "Identification Of Regression Models With A Misclassified And Endogenous Binary Regressor," Econometric Theory, Cambridge University Press, vol. 38(6), pages 1117-1139, December.
    20. Lorenzo Almada & Ian McCarthy & Rusty Tchernis, 2016. "What Can We Learn about the Effects of Food Stamps on Obesity in the Presence of Misreporting?," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 98(4), pages 997-1017.

    More about this item

    Keywords

    Corrupt sampling; Measurement error; Nonparametric bounds; Partial identification;
    All these keywords.

    JEL classification:

    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fbk:wpaper:2014-05. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Alessio Tomelleri or Daniela Anesi (email available below). General contact details of provider: https://edirc.repec.org/data/irvapit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.