IDEAS home Printed from https://ideas.repec.org/p/ucl/cepeow/20-09.html
   My bibliography  Save this paper

Conditioning: How background variables can influence PISA scores

Author

Listed:
  • Laura Zieger

    (Department of Social Science, UCL Institute of Education, University College London)

  • John Jerrim

    (Department of Social Science, UCL Institute of Education, University College London)

  • Jake Anders

    (Centre for Education Policy and Equalising Opportunities, UCL Institute of Education, University College London)

  • Nikki Shure

    (Department of Social Science, UCL Institute of Education, University College London)

Abstract

The Programme for International Student Assessment (PISA) is an international large-scale assessment which examines the educational achievement of 15-year-old students across the world. It has long become one of the key studies for evidence-based education policymaking across the globe. As result, PISA results and the methodology that they are based on should be robust, open and transparent. Yet, PISA receives significant criticism for its scaling model and the opaqueness in communicating it. One particular point of concern is the so-called "conditioning model", where background variables are used in the derivation of student achievement scores. The aim of this paper is to investigate this part of the scaling model and the impact it has upon the final scores. This includes varying the background variables of the conditioning model systematically and analysing the impact that this has on multiple measures. Our key finding is that the exact specification of the conditioning model matters and has substantial impact on average scores in some of the minor PISA domains (namely reading). It also has a major impact upon cross-national comparisons of educational inequality.

Suggested Citation

  • Laura Zieger & John Jerrim & Jake Anders & Nikki Shure, 2020. "Conditioning: How background variables can influence PISA scores," CEPEO Working Paper Series 20-09, UCL Centre for Education Policy and Equalising Opportunities, revised Apr 2020.
  • Handle: RePEc:ucl:cepeow:20-09
    as

    Download full text from publisher

    File URL: https://repec-cepeo.ucl.ac.uk/cepeow/cepeowp20-09.pdf
    File Function: First version, 2020
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Pan, Jianxin & Thompson, Robin, 2007. "Quasi-Monte Carlo estimation in generalized linear mixed models," Computational Statistics & Data Analysis, Elsevier, vol. 51(12), pages 5765-5775, August.
    2. John Micklewright & Sylke V. Schnepf & Chris Skinner, 2012. "Non-response biases in surveys of schoolchildren: the case of the English Programme for International Student Assessment (PISA) samples," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(4), pages 915-938, October.
    3. Adriana Ferro & Pedro Freitas & Luis Catela Nunes & Ana Balcao Reis & Carmo Seabra, 2015. "Correcting for sampling problems in PISA and the improvement in Portuguese students' performance," Nova SBE Working Paper Series wp591, Universidade Nova de Lisboa, Nova School of Business and Economics.
    4. Robert Mislevy, 1991. "Randomization-based inference about latent variables from complex samples," Psychometrika, Springer;The Psychometric Society, vol. 56(2), pages 177-196, June.
    5. Svend Kreiner & Karl Christensen, 2014. "Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 210-231, April.
    6. Gamboa, Luis Fernando & Waltenberg, Fábio D., 2012. "Inequality of opportunity for educational achievement in Latin America: Evidence from PISA 2006–2009," Economics of Education Review, Elsevier, vol. 31(5), pages 694-708.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maarten Marsman & Gunter Maris & Timo Bechger & Cees Glas, 2016. "What can we learn from Plausible Values?," Psychometrika, Springer;The Psychometric Society, vol. 81(2), pages 274-289, June.
    2. Liu, Ji & Steiner-Khamsi, Gita, 2020. "Human Capital Index and the hidden penalty for non-participation in ILSAs," International Journal of Educational Development, Elsevier, vol. 73(C).
    3. Gabriele B. Durrant & Sylke V. Schnepf, 2018. "Which schools and pupils respond to educational achievement surveys?: a focus on the English Programme for International Student Assessment sample," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1057-1075, October.
    4. Agasisti, Tommaso & de Oliveira Ribeiro, Celma & Montemor, Daniel Sanches, 2022. "The efficiency of Brazilian elementary public schools," International Journal of Educational Development, Elsevier, vol. 93(C).
    5. Francisco H. G. Ferreira & Jérémie Gignoux, 2014. "The Measurement of Educational Inequality: Achievement and Opportunity," The World Bank Economic Review, World Bank Group, vol. 28(2), pages 210-246.
    6. Natalia Kruger & Luis Fernando Gamboa & Fábio Waltenberg, 2014. "Gross Inequality and Inequality of Opportunities in Basic Education: Were they affected by Latin America’s Economic Boom?," Documentos de Trabajo 12322, Universidad del Rosario.
    7. Curt Hagquist & Raili Välimaa & Nina Simonsen & Sakari Suominen, 2017. "Differential Item Functioning in Trend Analyses of Adolescent Mental Health – Illustrative Examples Using HBSC-Data from Finland," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 10(3), pages 673-691, September.
    8. Simon Grund & Oliver Lüdtke & Alexander Robitzsch, 2018. "Multiple Imputation of Missing Data at Level 2: A Comparison of Fully Conditional and Joint Modeling in Multilevel Designs," Journal of Educational and Behavioral Statistics, , vol. 43(3), pages 316-353, June.
    9. Jakubowski, Maciej & Pokropek, Artur, 2015. "Reading achievement progress across countries," International Journal of Educational Development, Elsevier, vol. 45(C), pages 77-88.
    10. Luis Fernando Gamboa & Erika Londoño, 2015. "Assessing Educational Unfair Inequalities at a Regional Level in Colombia," Lecturas de Economía, Universidad de Antioquia, Departamento de Economía, issue 83, pages 97-133, Julio - D.
    11. Engzell, Per, 2017. "What Do Books in the Home Proxy For? A Cautionary Tale," Working Paper Series 1/2016, Stockholm University, Swedish Institute for Social Research.
    12. John Jerrim & Anna Vignoles & Ross Finnie, 2012. "University access for disadvantaged children: A comparison across English speaking countries," DoQSS Working Papers 12-11, Quantitative Social Science - UCL Social Research Institute, University College London.
    13. Camarero Garcia, Sebastian, 2022. "Inequality of Educational Opportunities and the Role of Learning Intensity," Labour Economics, Elsevier, vol. 74(C).
    14. Lire Ersado & Jérémie Gignoux, 2017. "Egypt: inequality of opportunity in education," Middle East Development Journal, Taylor & Francis Journals, vol. 9(1), pages 22-54, January.
    15. José María Rentería, 2023. "Inequality of Educational Opportunity and Time-Varying Circumstances: Longitudinal Evidence from Peru," Journal of Development Studies, Taylor & Francis Journals, vol. 59(2), pages 258-278, February.
    16. Robitzsch, Alexander, 2020. "About Still Nonignorable Consequences of (Partially) Ignoring Missing Item Responses in Large-scale Assessment," OSF Preprints hmy45, Center for Open Science.
    17. Lire Ersado & Jérémie Gignoux, 2014. "Inequality of Educational Opportunities in Egypt," PSE Working Papers halshs-01064510, HAL.
    18. Ammermüller, Andreas & Dolton, Peter J., 2006. "Pupil-teacher gender interaction effects on scholastic outcomes in England and the USA," ZEW Discussion Papers 06-060, ZEW - Leibniz Centre for European Economic Research.
    19. Márcia de Carvalho & Luis Fernando Gamboa & Fábio D. Waltenberg, 2012. "Equality of educational opportunity employing PISA data: taking both achievement and access into account," Documentos de Trabajo 10239, Universidad del Rosario.
    20. Giulia Grisolia & Umberto Lucia & Marco Filippo Torchio, 2022. "Sustainable Development and Workers Ability: Considerations on the Education Index in the Human Development Index," Sustainability, MDPI, vol. 14(14), pages 1-18, July.

    More about this item

    Keywords

    PISA; conditioning model; research transparency; scaling model; cross-national comparison; educational performance; educational inequality; test scores;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • I20 - Health, Education, and Welfare - - Education - - - General
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I24 - Health, Education, and Welfare - - Education - - - Education and Inequality

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ucl:cepeow:20-09. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jake Anders (email available below). General contact details of provider: https://edirc.repec.org/data/epucluk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.