IDEAS home Printed from https://ideas.repec.org/p/jrs/wpaper/201805.html
   My bibliography  Save this paper

Insights into survey errors of large scale educational achievement surveys

Author

Abstract

While educational achievement surveys revolutionised research on education cross-nationally, the surveys have been repeatedly subject of heated debate since first results were published. This paper reviews existing research examining the design and methodology of educational achievement surveys. Results are reported by allocating them to the specific survey error component of achievement estimates they address. Different error components from the design, collection, processing and analysis of survey data constitute the total survey error, which is an error difficult to quantify but important for assessing the overall accuracy of the surveys’ achievement estimates. The review shows that there are many reasons to assume that the total survey error associated with countries’ educational achievement estimates is likely to be inflated by other errors besides the standard error reported by survey organisers. Given the policy relevance of the surveys’ estimates, policy makers and the research community would greatly benefit from survey organisers providing more transparency on the different potential errors of educational achievement estimates. Without this information the debate about the fitness of educational achievement data for policy making is unlikely to dissolve.

Suggested Citation

  • Schnepf, Sylke, 2018. "Insights into survey errors of large scale educational achievement surveys," Working Papers 2018-05, Joint Research Centre, European Commission (Ispra site).
  • Handle: RePEc:jrs:wpaper:201805
    as

    Download full text from publisher

    File URL: http://publications.jrc.ec.europa.eu/repository/bitstream/JRC111734/jrc111734_jrc_wpef_insights_into_survey_errors_final_september_2018.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andreas Ammermueller & Jörn-Steffen Pischke, 2009. "Peer Effects in European Primary Schools: Evidence from the Progress in International Reading Literacy Study," Journal of Labor Economics, University of Chicago Press, vol. 27(3), pages 315-348, July.
    2. Nicole Schneeweis & Rudolf Winter-Ebmer, 2008. "Peer effects in Austrian schools," Studies in Empirical Economics, in: Christian Dustmann & Bernd Fitzenberger & Stephen Machin (ed.), The Economics of Education and Training, pages 133-155, Springer.
    3. Jerrim, John & Lopez-Agudo, Luis Alejandro & Marcenaro-Gutierrez, Oscar D. & Shure, Nikki, 2017. "What happens when econometrics and psychometrics collide? An example using the PISA data," Economics of Education Review, Elsevier, vol. 61(C), pages 51-58.
    4. Svend Kreiner & Karl Christensen, 2014. "Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 210-231, April.
    5. José M. Cordero & Víctor Cristóbal & Daniel Santín, 2018. "Causal Inference On Education Policies: A Survey Of Empirical Studies Using Pisa, Timss And Pirls," Journal of Economic Surveys, Wiley Blackwell, vol. 32(3), pages 878-915, July.
    6. Brian Jacob & Jesse Rothstein, 2016. "The Measurement of Student Ability in Modern Assessment Systems," Journal of Economic Perspectives, American Economic Association, vol. 30(3), pages 85-108, Summer.
    7. Nicholas Spaull, 2017. "Who Makes It Into PISA?: Understanding the Impact of PISA Sample Eligibility Using Turkey as a Case Study (PISA 2003 - PISA 2012)," OECD Education Working Papers 154, OECD Publishing.
    8. Giorgina Brown & John Micklewright & Sylke V. Schnepf & Robert Waldmann, 2007. "International surveys of educational achievement: how robust are the findings?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(3), pages 623-646, July.
    9. Micklewright, John & Schnepf, Sylke V. & Silva, Pedro N., 2012. "Peer effects and measurement error: The impact of sampling variation in school survey data (evidence from PISA)," Economics of Education Review, Elsevier, vol. 31(6), pages 1136-1142.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Hongliang, 2016. "The role of testing noise in the estimation of achievement-based peer effects," Economics of Education Review, Elsevier, vol. 54(C), pages 113-123.
    2. John Jerrim & Álvaro Choi, 2013. "The mathematics skills of school children: how does England compare to the high performing east Asian jurisdictions?," Working Papers 2013/12, Institut d'Economia de Barcelona (IEB).
    3. Micklewright, John & Schnepf, Sylke V. & Silva, Pedro N., 2012. "Peer effects and measurement error: The impact of sampling variation in school survey data (evidence from PISA)," Economics of Education Review, Elsevier, vol. 31(6), pages 1136-1142.
    4. Pedro N. Silva & John Micklewright & Sylke V. Schnepf, 2012. "The impact of sampling variation on peer measures: a comment on a proposal to adjust estimates for measurement error," DoQSS Working Papers 12-12, Quantitative Social Science - UCL Social Research Institute, University College London.
    5. Engzell, Per, 2017. "What Do Books in the Home Proxy For? A Cautionary Tale," Working Paper Series 1/2016, Stockholm University, Swedish Institute for Social Research.
    6. Ronny Freier & Johanna Storck, 2012. "The Treatment Effect of Attending a High-Quality School and the Influence of Unobservables," Discussion Papers of DIW Berlin 1256, DIW Berlin, German Institute for Economic Research.
    7. Frattini, Tommaso & Meschi, Elena, 2019. "The effect of immigrant peers in vocational schools," European Economic Review, Elsevier, vol. 113(C), pages 1-22.
    8. Ryan Yeung & Phuong Nguyen-Hoang, 2016. "Endogenous peer effects: Fact or fiction?," The Journal of Educational Research, Taylor & Francis Journals, vol. 109(1), pages 37-49, January.
    9. Oliver Falck & Stephan Heblich & Elke Luedemann, 2012. "Identity and entrepreneurship: do school peers shape entrepreneurial intentions?," Small Business Economics, Springer, vol. 39(1), pages 39-59, July.
    10. Olivier Monso & Denis Fougere & Pauline Givord & Claudine Pirus, 2019. "Les camarades influencent-ils la réussite et le parcours des élèves ? Une revue de littérature sur les effets de pairs dans l’enseignement primaire et secondaire," Sciences Po publications 86, Sciences Po.
    11. John Jerrim & Álvaro Choi, 2013. "The mathematics skills of school children: how does England compare to the high performing east Asian jurisdictions?," Working Papers 2013/12, Institut d'Economia de Barcelona (IEB).
    12. Elke Lüdemann, 2011. "Schooling and the Formation of Cognitive and Non-cognitive Outcomes," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 39, May.
    13. John Jerrim & Alvaro Choi, 2013. "The mathematics skills of school children: How does England compare to the high performing East Asian jurisdictions?," DoQSS Working Papers 13-03, Quantitative Social Science - UCL Social Research Institute, University College London.
    14. Liu, Ji & Steiner-Khamsi, Gita, 2020. "Human Capital Index and the hidden penalty for non-participation in ILSAs," International Journal of Educational Development, Elsevier, vol. 73(C).
    15. Vardardottir, Arna, 2015. "The impact of classroom peers in a streaming system," Economics of Education Review, Elsevier, vol. 49(C), pages 110-128.
    16. Asako Ohinata & Jan C. van Ours, 2013. "How Immigrant Children Affect the Academic Achievement of Native Dutch Children," Economic Journal, Royal Economic Society, vol. 0, pages 308-331, August.
    17. Lang, Kevin & Siniver, Erez, 2011. "Why is an elite undergraduate education valuable? Evidence from Israel," Labour Economics, Elsevier, vol. 18(6), pages 767-777.
    18. Todd D. Gerarden & Richard G. Newell & Robert N. Stavins, 2017. "Assessing the Energy-Efficiency Gap," Journal of Economic Literature, American Economic Association, vol. 55(4), pages 1486-1525, December.
    19. Adam S. Booij & Edwin Leuven & Hessel Oosterbeek, 2017. "Ability Peer Effects in University: Evidence from a Randomized Experiment," Review of Economic Studies, Oxford University Press, vol. 84(2), pages 547-578.
    20. Ana Balsa & Néstor Gandelman & Flavia Roldán, 2017. "Peer and parental influence in the development of cognitive skills and predispostion to risky behaviour," Documentos de Trabajo/Working Papers 1701, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..

    More about this item

    Keywords

    educational achievement surveys; survey methodology; survey errors; PISA; TIMSS; PIRLS;
    All these keywords.

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jrs:wpaper:201805. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/eejrcit.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Benczur (email available below). General contact details of provider: https://edirc.repec.org/data/eejrcit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.