IDEAS home Printed from https://ideas.repec.org/p/jrs/wpaper/201805.html
   My bibliography  Save this paper

Insights into survey errors of large scale educational achievement surveys

Author

Abstract

While educational achievement surveys revolutionised research on education cross-nationally, the surveys have been repeatedly subject of heated debate since first results were published. This paper reviews existing research examining the design and methodology of educational achievement surveys. Results are reported by allocating them to the specific survey error component of achievement estimates they address. Different error components from the design, collection, processing and analysis of survey data constitute the total survey error, which is an error difficult to quantify but important for assessing the overall accuracy of the surveys’ achievement estimates. The review shows that there are many reasons to assume that the total survey error associated with countries’ educational achievement estimates is likely to be inflated by other errors besides the standard error reported by survey organisers. Given the policy relevance of the surveys’ estimates, policy makers and the research community would greatly benefit from survey organisers providing more transparency on the different potential errors of educational achievement estimates. Without this information the debate about the fitness of educational achievement data for policy making is unlikely to dissolve.

Suggested Citation

  • Schnepf, Sylke, 2018. "Insights into survey errors of large scale educational achievement surveys," Working Papers 2018-05, Joint Research Centre, European Commission.
    • Handle: RePEc:jrs:wpaper:201805
    as

    Download full text from publisher

    File URL: http://publications.jrc.ec.europa.eu/repository/bitstream/JRC111734/jrc111734_jrc_wpef_insights_into_survey_errors_final_september_2018.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andreas Ammermueller & Jörn-Steffen Pischke, 2009. "Peer Effects in European Primary Schools: Evidence from the Progress in International Reading Literacy Study," Journal of Labor Economics, University of Chicago Press, vol. 27(3), pages 315-348, July.
    2. Jerrim, John & Lopez-Agudo, Luis Alejandro & Marcenaro-Gutierrez, Oscar D. & Shure, Nikki, 2017. "What happens when econometrics and psychometrics collide? An example using the PISA data," Economics of Education Review, Elsevier, vol. 61(C), pages 51-58.
    3. Nicholas Spaull, 2017. "Who Makes It Into PISA?: Understanding the Impact of PISA Sample Eligibility Using Turkey as a Case Study (PISA 2003 - PISA 2012)," OECD Education Working Papers 154, OECD Publishing.
    4. Giorgina Brown & John Micklewright & Sylke V. Schnepf & Robert Waldmann, 2007. "International surveys of educational achievement: how robust are the findings?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(3), pages 623-646, July.
    5. Brian Jacob & Jesse Rothstein, 2016. "The Measurement of Student Ability in Modern Assessment Systems," Journal of Economic Perspectives, American Economic Association, vol. 30(3), pages 85-108, Summer.
    6. Nicole Schneeweis & Rudolf Winter-Ebmer, 2008. "Peer effects in Austrian schools," Studies in Empirical Economics, in: Christian Dustmann & Bernd Fitzenberger & Stephen Machin (ed.), The Economics of Education and Training, pages 133-155, Springer.
    7. Svend Kreiner & Karl Christensen, 2014. "Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 210-231, April.
    8. José M. Cordero & Víctor Cristóbal & Daniel Santín, 2018. "Causal Inference On Education Policies: A Survey Of Empirical Studies Using Pisa, Timss And Pirls," Journal of Economic Surveys, Wiley Blackwell, vol. 32(3), pages 878-915, July.
    9. Micklewright, John & Schnepf, Sylke V. & Silva, Pedro N., 2012. "Peer effects and measurement error: The impact of sampling variation in school survey data (evidence from PISA)," Economics of Education Review, Elsevier, vol. 31(6), pages 1136-1142.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vargas-Montoya, Luis & Gimenez, Gregorio & Fernández-Gutiérrez, Marcos, 2023. "ICT use for learning and students' outcomes: Does the country's development level matter?," Socio-Economic Planning Sciences, Elsevier, vol. 87(PA).
    2. Michele Raitano & Francesco Vona, 2013. "Peer heterogeneity, school tracking and students' performances: evidence from PISA 2006," Applied Economics, Taylor & Francis Journals, vol. 45(32), pages 4516-4532, November.
    3. Engzell, Per, 2017. "What Do Books in the Home Proxy For? A Cautionary Tale," Working Paper Series 1/2016, Stockholm University, Swedish Institute for Social Research.
    4. Ronny Freier & Johanna Storck, 2012. "The Treatment Effect of Attending a High-Quality School and the Influence of Unobservables," Discussion Papers of DIW Berlin 1256, DIW Berlin, German Institute for Economic Research.
    5. Michele Raitano & Francesco Vona, 2010. "Peer Heterogeneity, Parental Background and Tracking: Evidence from PISA 2006," Documents de Travail de l'OFCE 2010-23, Observatoire Francais des Conjonctures Economiques (OFCE).
    6. Frattini, Tommaso & Meschi, Elena, 2019. "The effect of immigrant peers in vocational schools," European Economic Review, Elsevier, vol. 113(C), pages 1-22.
    7. Olivier Monso & Denis Fougère & Pauline Givord & Claudine Pirus, 2019. "Les camarades influencent-ils la réussite et le parcours des élèves ? Une revue de littérature sur les effets de pairs dans l’enseignement primaire et secondaire," Working Papers hal-03456953, HAL.
    8. John Jerrim & Álvaro Choi, 2013. "The mathematics skills of school children: how does England compare to the high performing east Asian jurisdictions?," Working Papers 2013/12, Institut d'Economia de Barcelona (IEB).
    9. Elke Lüdemann, 2011. "Schooling and the Formation of Cognitive and Non-cognitive Outcomes," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 39.
    10. Cathles, Alison & Ou, Dongshu & Sasso, Simone & Setrana, Mary & van Veen, Tom, 2021. "Where do you come from, where do you go? Assessing skills gaps and labour market outcomes for young adults with different immigration backgrounds," International Journal of Educational Development, Elsevier, vol. 86(C).
    11. Schnepf, Sylke V. & Granato, Silvia, 2023. "COVID-19 and the European Education Performance Decline: A Focus on Primary School Children's Reading Achievement between 2016 and 2021," IZA Discussion Papers 16531, Institute of Labor Economics (IZA).
    12. Gregorio Gimenez & Denisa Ciobanu & Beatriz Barrado, 2021. "A Proposal of Spatial Measurement of Peer Effect through Socioeconomic Indices and Unsatisfied Basic Needs," Economies, MDPI, vol. 9(2), pages 1-20, May.
    13. Oliver Falck & Stephan Heblich & Elke Luedemann, 2012. "Identity and entrepreneurship: do school peers shape entrepreneurial intentions?," Small Business Economics, Springer, vol. 39(1), pages 39-59, July.
    14. José M. Cordero & Víctor Cristóbal & Daniel Santín, 2018. "Causal Inference On Education Policies: A Survey Of Empirical Studies Using Pisa, Timss And Pirls," Journal of Economic Surveys, Wiley Blackwell, vol. 32(3), pages 878-915, July.
    15. Vardardottir, Arna, 2015. "The impact of classroom peers in a streaming system," Economics of Education Review, Elsevier, vol. 49(C), pages 110-128.
    16. Zhang, Hongliang, 2016. "The role of testing noise in the estimation of achievement-based peer effects," Economics of Education Review, Elsevier, vol. 54(C), pages 113-123.
    17. John Jerrim & Álvaro Choi, 2013. "The mathematics skills of school children: how does England compare to the high performing east Asian jurisdictions?," Working Papers 2013/12, Institut d'Economia de Barcelona (IEB).
    18. Micklewright, John & Schnepf, Sylke V. & Silva, Pedro N., 2012. "Peer effects and measurement error: The impact of sampling variation in school survey data (evidence from PISA)," Economics of Education Review, Elsevier, vol. 31(6), pages 1136-1142.
    19. Pedro N. Silva & John Micklewright & Sylke V. Schnepf, 2012. "The impact of sampling variation on peer measures: a comment on a proposal to adjust estimates for measurement error," DoQSS Working Papers 12-12, Quantitative Social Science - UCL Social Research Institute, University College London.
    20. Ryan Yeung & Phuong Nguyen-Hoang, 2016. "Endogenous peer effects: Fact or fiction?," The Journal of Educational Research, Taylor & Francis Journals, vol. 109(1), pages 37-49, January.

    More about this item

    Keywords

    educational achievement surveys; survey methodology; survey errors; PISA; TIMSS; PIRLS;
    All these keywords.

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jrs:wpaper:201805. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Benczur (email available below). General contact details of provider: https://edirc.repec.org/data/eejrcit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.