IDEAS home Printed from https://ideas.repec.org/p/qss/dqsswp/2019.html
   My bibliography  Save this paper

From asking to observing. Behavioural measures of socio-emotional and motivational skills in large-scale assessments

Author

Listed:
  • Francesca Borgonovi

    (UCL Social Research Institute, United Kingdom)

  • Alessandro Ferrara

    (Department of Political and Social Sciences, European University Institute, Italy)

  • Mario Piacentini

    (Directorate for Education and Skills, France)

Abstract

Socio-emotional and motivational skills are routinely measured using self-reports in large-scale educational assessments. Measures exploiting test-takers’ behaviour during the completion of questionnaires or cognitive tests are increasingly used as alternatives to self-reports in the economics of education literature. We use cross-sectional and longitudinal evidence to evaluate if behavioural measures can provide unbiased measures of socio-emotional and motivational skills to be used in empirical research using data from the Programme for International Student Assessment (PISA). We find that behavioural measures capture important aspects of students’ academic profiles: some are importantly associated with contemporaneous performance and educational attainment. However, these measures are only limitedly correlated among themselves and have low correlations with self-report measures of the same constructs. Moreover, behavioural measures have different levels of stability over time and sensitivity to design considerations. These results suggest that more research is needed before measures of students’ behaviour on a cognitive test can be used as valid indicators of socio-emotional and motivational skills.

Suggested Citation

  • Francesca Borgonovi & Alessandro Ferrara & Mario Piacentini, 2020. "From asking to observing. Behavioural measures of socio-emotional and motivational skills in large-scale assessments," DoQSS Working Papers 20-19, Quantitative Social Science - UCL Social Research Institute, University College London.
    • Handle: RePEc:qss:dqsswp:2019
    as

    Download full text from publisher

    File URL: http://repec.ioe.ac.uk/REPEc/pdf/qsswp2019.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Falk, Armin & Hermle, Johannes, 2018. "Relationship of Gender Differences in Preferences to Economic Development and Gender Equality," IZA Discussion Papers 12059, Institute of Labor Economics (IZA).
    2. Uri Gneezy & John A. List & Jeffrey A. Livingston & Xiangdong Qin & Sally Sadoff & Yang Xu, 2019. "Measuring Success in Education: The Role of Effort on the Test Itself," American Economic Review: Insights, American Economic Association, vol. 1(3), pages 291-308, December.
    3. Tim Kautz & James J. Heckman & Ron Diris & Bas ter Weel & Lex Borghans, 2014. "Fostering and Measuring Skills: Improving Cognitive and Non-cognitive Skills to Promote Lifetime Success," OECD Education Working Papers 110, OECD Publishing.
    4. Pau Balart & Matthijs Oosterveen, 2019. "Females show more sustained performance during test-taking than males," Nature Communications, Nature, vol. 10(1), pages 1-11, December.
    5. Simon Breakspear, 2012. "The Policy Impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance," OECD Education Working Papers 71, OECD Publishing.
    6. Brunello, Giorgio & Crema, Angela & Rocco, Lorenzo, 2018. "Testing at Length If It Is Cognitive or Non-Cognitive," IZA Discussion Papers 11603, Institute of Labor Economics (IZA).
    7. Jeffrey Mo, 2017. "Collaborative problem solving," PISA in Focus 78, OECD Publishing.
    8. Pelin Akyol & Kala Krishna & Jinwen Wang, 2021. "Taking PISA Seriously: How Accurate are Low-Stakes Exams?," Journal of Labor Research, Springer, vol. 42(2), pages 184-243, June.
    9. Miloš Kankaraš, 2017. "Personality matters: Relevance and assessment of personality characteristics," OECD Education Working Papers 157, OECD Publishing.
    10. Trivitt, Julie & Cheng, Albert, 2016. "When you say nothing at all: The predictive power of student effort on surveysAuthor-Name: Hitt, Collin," Economics of Education Review, Elsevier, vol. 52(C), pages 105-119.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giorgio Brunello & Angela Crema & Lorenzo Rocco, 2021. "Some Unpleasant Consequences of Testing at Length," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 83(4), pages 1002-1023, August.
    2. Marianna Battaglia & Marisa Hidalgo-Hidalgo, 2020. "Non-Cognitive Skills and Remedial Education: Good News for Girls," Working Papers 20.10, Universidad Pablo de Olavide, Department of Economics.
    3. Dorner, Zack & Lancsar, Emily, 2023. "Don’t pay the highly motivated too much," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 103(C).
    4. de Hoyos, Rafael & Estrada, Ricardo & Vargas, María José, 2021. "What do test scores really capture? Evidence from a large-scale student assessment in Mexico," World Development, Elsevier, vol. 146(C).
    5. Filmer, Deon & Rogers, Halsey & Angrist, Noam & Sabarwal, Shwetlena, 2020. "Learning-adjusted years of schooling (LAYS): Defining a new macro measure of education," Economics of Education Review, Elsevier, vol. 77(C).
    6. Haeckl, Simone & Kartal, Melis, 2021. "Does a stereotype benefit women in the labor market: An experiment on perseverance," UiS Working Papers in Economics and Finance 2021/5, University of Stavanger.
    7. Clotilde Napp & Thomas Breda, 2022. "The stereotype that girls lack talent: A worldwide investigation," Post-Print halshs-03672465, HAL.
    8. Anghel, Brindusa & Rodríguez-Planas, Núria & Sanz-de-Galdeano, Anna, 2020. "Is the math gender gap associated with gender equality? Only in low-income countries," Economics of Education Review, Elsevier, vol. 79(C).
    9. Pelin Akyol, 2021. "Comparison of Computer-based and Paper-based Exams: Evidence from PISA," Bogazici Journal, Review of Social, Economic and Administrative Studies, Bogazici University, Department of Economics, vol. 35(2), pages 137-150.
    10. Hai-Anh Dang & Paul Glewwe & Khoa Vu & Jongwook Lee, 2021. "What Explains Vietnam’s Exceptional Performance in Education Relative to Other Countries? Analysis of the 2012 and 2015 PISA Data," Working Papers 580, ECINEQ, Society for the Study of Economic Inequality.
    11. Brunello, Giorgio & Kiss, David, 2022. "Math scores in high stakes grades," Economics of Education Review, Elsevier, vol. 87(C).
    12. Silvia Griselda, 2020. "Different Questions, Different Gender Gap: Can the Format of Questions Explain the Gender Gap in Mathematics?," 2020 Papers pgr710, Job Market Papers.
    13. Clotilde Napp & Thomas Breda, 2022. "The stereotype that girls lack talent: A worldwide investigation," PSE-Ecole d'économie de Paris (Postprint) halshs-03672465, HAL.
    14. Germán Reyes, 2023. "Cognitive Endurance, Talent Selection, and the Labor Market Returns to Human Capital," CRC TR 224 Discussion Paper Series crctr224_2023_490, University of Bonn and University of Mannheim, Germany.
    15. Germ'an Reyes, 2023. "Cognitive Endurance, Talent Selection, and the Labor Market Returns to Human Capital," Papers 2301.02575, arXiv.org.
    16. Sonja C. Kassenboehmer & Stefanie Schurer, 2018. "Survey item-response behavior as an imperfect proxy for unobserved ability: Theory and application," Working Papers 2018-035, Human Capital and Economic Opportunity Working Group.
    17. Dang, Hai-Anh & Glewwe, Paul & Vu, Khoa & Lee, Jongwook, 2021. "What Explains Vietnam's Exceptional Performance in Education Relative to Other Countries? Analysis of the 2012 and 2015 Pisa Data," IZA Discussion Papers 14315, Institute of Labor Economics (IZA).
    18. Sharafi, Zahra, 2023. "Poverty and perseverance: The detrimental effect of poverty on effort provision," Journal of Development Economics, Elsevier, vol. 162(C).
    19. Lucas Gortazar, 2019. "¿Favorece el sistema educativo español la igualdad de oportunidades?," Studies on the Spanish Economy eee2019-17, FEDEA.
    20. Elizabeth M. Caucutt & Lance Lochner & Youngmin Park, 2017. "Correlation, Consumption, Confusion, or Constraints: Why Do Poor Children Perform so Poorly?," Scandinavian Journal of Economics, Wiley Blackwell, vol. 119(1), pages 102-147, January.

    More about this item

    Keywords

    Socio-emotional and motivational skills; cross-country; PISA; large-scale assessments; behavioural measures; self-reports; education.;
    All these keywords.

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • I24 - Health, Education, and Welfare - - Education - - - Education and Inequality
    • I26 - Health, Education, and Welfare - - Education - - - Returns to Education

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:qss:dqsswp:2019. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Dr Neus Bover Fonts (email available below). General contact details of provider: https://edirc.repec.org/data/dqioeuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.