IDEAS home Printed from https://ideas.repec.org/a/eee/soceco/v72y2018icp51-60.html
   My bibliography  Save this article

Comparing and validating measures of non-cognitive traits: Performance task measures and self-reports from a nationally representative internet panel

Author

Listed:
  • Zamarro, Gema
  • Cheng, Albert
  • Shakeel, M. Danish
  • Hitt, Collin

Abstract

Education researchers and labor economists are increasingly focusing on skills such as conscientiousness, grit and emotional stability. This is part of a larger research program that sees “non-cognitive skills” as drivers of educational attainment and labor market outcomes. However, data on these important non-cognitive traits is not always available and, when available, researchers have raised concerns on potential biases in self-reported measures. In this paper, we use data collected from the Understanding America Study, a nationally representative internet panel, to study the validity of measures of non-cognitive traits, including novel measures based on survey effort. We propose that surveys can be viewed as performance tasks and that respondents provide meaningful information about relevant non-cognitive traits based on the effort they put forward in completing them. In particular, we examine the extent to which respondents provide careless answers or intentionally skip questions. We compare self-reported measures of grit, conscientiousness, neuroticism, and other personality traits, with survey-effort measures. We study the relationship of survey effort measures to these self-reported measures, educational attainment, and labor-market outcomes. Our results show that measures of careless answering in surveys show promise for use as proxies of traits related to conscientiousness and neuroticism.

Suggested Citation

  • Zamarro, Gema & Cheng, Albert & Shakeel, M. Danish & Hitt, Collin, 2018. "Comparing and validating measures of non-cognitive traits: Performance task measures and self-reports from a nationally representative internet panel," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 72(C), pages 51-60.
    • Handle: RePEc:eee:soceco:v:72:y:2018:i:c:p:51-60
    DOI: 10.1016/j.socec.2017.11.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2214804317301349
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.socec.2017.11.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Eric Hanushek & Ludger Woessmann, 2012. "Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation," Journal of Economic Growth, Springer, vol. 17(4), pages 267-321, December.
    2. Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), 2011. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 4, number 4, June.
    3. Kahneman, Daniel & Knetsch, Jack L & Thaler, Richard H, 1986. "Fairness and the Assumptions of Economics," The Journal of Business, University of Chicago Press, vol. 59(4), pages 285-300, October.
    4. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    5. Lex Borghans & Angela Lee Duckworth & James J. Heckman & Bas ter Weel, 2008. "The Economics and Psychology of Personality Traits," Journal of Human Resources, University of Wisconsin Press, vol. 43(4).
    6. James J. Heckman & Jora Stixrud & Sergio Urzua, 2006. "The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 411-482, July.
    7. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    8. Carmit Segal, 2012. "Working When No One Is Watching: Motivation, Test Scores, and Economic Success," Management Science, INFORMS, vol. 58(8), pages 1438-1457, August.
    9. Andreoni, James, 1995. "Cooperation in Public-Goods Experiments: Kindness or Confusion?," American Economic Review, American Economic Association, vol. 85(4), pages 891-904, September.
    10. Almlund, Mathilde & Duckworth, Angela Lee & Heckman, James & Kautz, Tim, 2011. "Personality Psychology and Economics," Handbook of the Economics of Education, in: Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), Handbook of the Economics of Education, edition 1, volume 4, chapter 0, pages 1-181, Elsevier.
    11. Bas ter Weel, 2008. "The Noncognitive Determinants of Labor Market and Behavioral Outcomes: Introduction to the Symposium," Journal of Human Resources, University of Wisconsin Press, vol. 43(4).
    12. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    13. Trivitt, Julie & Cheng, Albert, 2016. "When you say nothing at all: The predictive power of student effort on surveysAuthor-Name: Hitt, Collin," Economics of Education Review, Elsevier, vol. 52(C), pages 105-119.
    14. Deck, Cary & Lee, Jungmin & Reyes, Javier A. & Rosen, Christopher C., 2013. "A failed attempt to explain within subject variation in risk taking behavior using domain specific risk attitudes," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 1-24.
    15. Will Dobbie & Roland G. Fryer Jr., 2015. "The Medium-Term Impacts of High-Achieving Charter Schools," Journal of Political Economy, University of Chicago Press, vol. 123(5), pages 985-1037.
    16. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    17. Isaac M. Lipkus & Greg Samsa & Barbara K. Rimer, 2001. "General Performance on a Numeracy Scale among Highly Educated Samples," Medical Decision Making, , vol. 21(1), pages 37-44, February.
    18. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    19. Nicholas Bardsley, 2008. "Dictator game giving: altruism or artefact?," Experimental Economics, Springer;Economic Science Association, vol. 11(2), pages 122-133, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cheng, Albert & Zamarro, Gema, 2018. "Measuring teacher non-cognitive skills and its impact on students: Insight from the Measures of Effective Teaching Longitudinal Database," Economics of Education Review, Elsevier, vol. 64(C), pages 251-260.
    2. Fabian T C Schmidt & Clemens M Lechner & Daniel Danner, 2020. "New wine in an old bottle? A facet-level perspective on the added value of Grit over BFI–2 Conscientiousness," PLOS ONE, Public Library of Science, vol. 15(2), pages 1-25, February.
    3. Marianna Battaglia & Marisa Hidalgo-Hidalgo, 2020. "Non-Cognitive Skills and Remedial Education: Good News for Girls," Working Papers 20.10, Universidad Pablo de Olavide, Department of Economics.
    4. Albert Cheng & Gema Zamarro & Bart Orriens, 2020. "Personality as a Predictor of Unit Nonresponse in an Internet Panel," Sociological Methods & Research, , vol. 49(3), pages 672-698, August.
    5. Anna Bussu & Dimitri Paolini & Manuela Pulina & Giuseppe Zanzurino, 2023. "From Choice to Performance in Secondary Schools: Evidence from a Disadvantaged Setting in Italy," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 9(2), pages 529-555, July.
    6. Esther Ulitzsch & Qiwei He & Vincent Ulitzsch & Hendrik Molter & André Nichterlein & Rolf Niedermeier & Steffi Pohl, 2021. "Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 190-214, March.
    7. Zhang, Yi & He, Jia, 2023. "Measuring non-cognitive skills exploiting log-files on online behaviour," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 103(C).
    8. Germán Reyes, 2023. "Cognitive Endurance, Talent Selection, and the Labor Market Returns to Human Capital," CRC TR 224 Discussion Paper Series crctr224_2023_490, University of Bonn and University of Mannheim, Germany.
    9. Germ'an Reyes, 2023. "Cognitive Endurance, Talent Selection, and the Labor Market Returns to Human Capital," Papers 2301.02575, arXiv.org.
    10. Gema Zamarro & Malachi Nichols & Angela L Duckworth & Sidney K D’Mello, 2020. "Validation of survey effort measures of grit and self-control in a sample of high school students," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-16, July.
    11. María Ladrón de Guevara Rodríguez & Oscar David Marcenaro-Gutierrez & Luis Alejandro Lopez-Agudo, 2023. "On the Gender Gap of Soft-Skills: the Spanish Case," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 16(1), pages 167-197, February.
    12. Marcenaro-Gutierrez, O.D. & Lopez-Agudo, L.A. & Henriques, C.O., 2021. "Are soft skills conditioned by conflicting factors? A multiobjective programming approach to explore the trade-offs," Economic Analysis and Policy, Elsevier, vol. 72(C), pages 18-40.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    2. Balart, Pau & Oosterveen, Matthijs & Webbink, Dinand, 2018. "Test scores, noncognitive skills and economic growth," Economics of Education Review, Elsevier, vol. 63(C), pages 134-153.
    3. James J. Heckman & Tomáš Jagelka & Timothy D. Kautz, 2019. "Some Contributions of Economics to the Study of Personality," NBER Working Papers 26459, National Bureau of Economic Research, Inc.
    4. Trivitt, Julie & Cheng, Albert, 2016. "When you say nothing at all: The predictive power of student effort on surveysAuthor-Name: Hitt, Collin," Economics of Education Review, Elsevier, vol. 52(C), pages 105-119.
    5. Dasgupta, Utteeyo & Gangadharan, Lata & Maitra, Pushkar & Mani, Subha, 2017. "Searching for preference stability in a state dependent world," Journal of Economic Psychology, Elsevier, vol. 62(C), pages 17-32.
    6. Peter Hoeschler & Uschi Backes-Gellner, 2017. "The Relative Importance of Personal Characteristics for the Hiring of Young Workers," Economics of Education Working Paper Series 0142, University of Zurich, Department of Business Administration (IBW), revised Jan 2018.
    7. Ben Weidmann & David J. Deming, 2020. "Team Players: How Social Skills Improve Group Performance," NBER Working Papers 27071, National Bureau of Economic Research, Inc.
    8. Kautz, Tim & Heckman, James J. & Diris, Ron & ter Weel, Bas & Borghans, Lex, 2014. "Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success," IZA Discussion Papers 8696, Institute of Labor Economics (IZA).
    9. María José Roa & Ignacio Garrón & Jonathan Barboza, 2019. "Financial Decisions and Financial Capabilities in the Andean Region," Journal of Consumer Affairs, Wiley Blackwell, vol. 53(2), pages 296-323, June.
    10. Ondrej Rydval, 2012. "The Causal Effect of Cognitive Abilities on Economic Behavior: Evidence from a Forecasting Task with Varying Cognitive Load," Jena Economics Research Papers 2011-064, Friedrich-Schiller-University Jena.
    11. Balsa, Ana & Gandelman, Néstor & Roldán, Flavia, 2015. "Peer Effects in the Development of Capabilities in Adolescence," Research Department working papers 820, CAF Development Bank Of Latinamerica.
    12. Maria Cubel & Ana Nuevo‐Chiquero & Santiago Sanchez‐Pages & Marian Vidal‐Fernandez, 2016. "Do Personality Traits Affect Productivity? Evidence from the Laboratory," Economic Journal, Royal Economic Society, vol. 0(592), pages 654-681, May.
    13. Germ'an Reyes, 2023. "Cognitive Endurance, Talent Selection, and the Labor Market Returns to Human Capital," Papers 2301.02575, arXiv.org.
    14. Marianna Battaglia & Marisa Hidalgo-Hidalgo, 2020. "Non-Cognitive Skills and Remedial Education: Good News for Girls," Working Papers 20.10, Universidad Pablo de Olavide, Department of Economics.
    15. Jonathan P. Beauchamp & David Cesarini & Magnus Johannesson, 2017. "The psychometric and empirical properties of measures of risk preferences," Journal of Risk and Uncertainty, Springer, vol. 54(3), pages 203-237, June.
    16. Goel, Deepti & Barooah, Bidisha, 2018. "Drivers of Student Performance: Evidence from Higher Secondary Public Schools in Delhi," GLO Discussion Paper Series 231, Global Labor Organization (GLO).
    17. Borghans, Lex & Meijers, Huub & ter Weel, Bas, 2013. "The importance of intrinsic and extrinsic motivation for measuring IQ," Economics of Education Review, Elsevier, vol. 34(C), pages 17-28.
    18. Beattie, Graham & Laliberté, Jean-William P. & Oreopoulos, Philip, 2018. "Thrivers and divers: Using non-academic measures to predict college success and failure," Economics of Education Review, Elsevier, vol. 62(C), pages 170-182.
    19. Mendolia, Silvia & Walker, Ian, 2014. "The effect of personality traits on subject choice and performance in high school: Evidence from an English cohort," Economics of Education Review, Elsevier, vol. 43(C), pages 47-65.
    20. Brice Corgnet & Cary Deck & Mark DeSantis & David Porter, 2022. "Forecasting Skills in Experimental Markets: Illusion or Reality?," Management Science, INFORMS, vol. 68(7), pages 5216-5232, July.

    More about this item

    Keywords

    Non-cognitive traits; Grit; Conscientiousness; Performance tasks; Survey effort;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:soceco:v:72:y:2018:i:c:p:51-60. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/620175 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.