IDEAS home Printed from https://ideas.repec.org/a/eee/soceco/v82y2019ics2214804319301569.html

Cognitive reflection test: Whom, how, when

Author

Listed:
  • Brañas-Garza, Pablo
  • Kujal, Praveen
  • Lenkei, Balint

Abstract

The use of the Cognitive Reflection Test as a covariate to explain behavior in Economics and Psychology experiments has significantly increased in the past few years. Experiments have shown its usefulness in predicting behavior. However, little is known about if the test is gender biased, whether incentives matter or how different implementation procedures impact outcomes. Here we report the results of a meta-study of 118 Cognitive Reflection Test studies comprising of 44,558 participants across 21 countries. We find that there is a negative correlation between being female and the overall, and individual, correct answers to CRT questions. Monetary incentives do not impact performance. Regarding implementation procedures, taking the test at the end of the experiment negatively impacts performance. Students perform better compared to non-students. We obtain mixed evidence on whether the sequence of questions matters. Finally, we find that computerized tests marginally improve results.

Suggested Citation

  • Brañas-Garza, Pablo & Kujal, Praveen & Lenkei, Balint, 2019. "Cognitive reflection test: Whom, how, when," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 82(C).
    • Handle: RePEc:eee:soceco:v:82:y:2019:i:c:s2214804319301569
    DOI: 10.1016/j.socec.2019.101455
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2214804319301569
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.socec.2019.101455?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or

    for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:hal:pseose:hal-01359231 is not listed on IDEAS
    2. Andersson, Ola & Tyran, Jean-Robert & Wengström, Erik & Holm, Håkan J., 2013. "Risk Aversion Relates to Cognitive Ability: Fact or Fiction?," Working Paper Series 964, Research Institute of Industrial Economics.
    3. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    4. Brañas-Garza, Pablo & García-Muñoz, Teresa & González, Roberto Hernán, 2012. "Cognitive effort in the Beauty Contest Game," Journal of Economic Behavior & Organization, Elsevier, vol. 83(2), pages 254-260.
    5. Besedeš, Tibor & Deck, Cary & Sarangi, Sudipta & Shor, Mikhael, 2012. "Decision-making strategies and performance among seniors," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 524-533.
    6. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    7. Ola Andersson & Håkan J. Holm & Jean-Robert Tyran & Erik Wengström, 2016. "Risk Aversion Relates to Cognitive Ability: Preferences Or Noise?," Journal of the European Economic Association, European Economic Association, vol. 14(5), pages 1129-1154.
    8. Brice Corgnet & Roberto Hernán-González & Praveen Kujal & David Porter, 2015. "The Effect of Earned Versus House Money on Price Bubble Formation in Experimental Asset Markets," Review of Finance, European Finance Association, vol. 19(4), pages 1455-1488.
    9. Carpenter, Jeffrey & Graham, Michael & Wolf, Jesse, 2013. "Cognitive ability and strategic sophistication," Games and Economic Behavior, Elsevier, vol. 80(C), pages 115-130.
    10. Bonner, Sarah E. & Sprinkle, Geoffrey B., 2002. "The effects of monetary incentives on effort and task performance: theories, evidence, and a framework for research," Accounting, Organizations and Society, Elsevier, vol. 27(4-5), pages 303-345.
    11. Douglas W. Diamond & Philip H. Dybvig, 2000. "Bank runs, deposit insurance, and liquidity," Quarterly Review, Federal Reserve Bank of Minneapolis, vol. 24(Win), pages 14-23.
    12. Cueva, Carlos & Iturbe-Ormaetxe, Iñigo & Mata-Pérez, Esther & Ponti, Giovanni & Sartarelli, Marcello & Yu, Haihan & Zhukova, Vita, 2016. "Cognitive (ir)reflection: New experimental evidence," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 81-93.
    13. Charles N. Noussair & Stefan T. Trautmann & Gijs van de Kuilen, 2014. "Higher Order Risk Attitudes, Demographics, and Financial Decisions," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 325-355.
    14. Ben-Ner, Avner & Kong, Fanmin & Putterman, Louis, 2004. "Share and share alike? Gender-pairing, personality, and cognitive ability as determinants of giving," Journal of Economic Psychology, Elsevier, vol. 25(5), pages 581-589, October.
    15. Guillermo Campitelli & Martin Labollita, 2010. "Correlations of cognitive reflection with judgments and choices," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(3), pages 182-191, June.
    16. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    17. Ruffle, Bradley J. & Wilson, Anne E., 2018. "The truth about tattoos," Economics Letters, Elsevier, vol. 172(C), pages 143-147.
    18. Cheung, Stephen L. & Hedegaard, Morten & Palan, Stefan, 2014. "To see is to believe: Common expectations in experimental asset markets," European Economic Review, Elsevier, vol. 66(C), pages 84-96.
    19. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    20. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    21. Michael N. Stagnaro & Gordon Pennycook & David G. Rand, 2018. "Performance on the Cognitive Reflection Test is stable across time," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(3), pages 260-267, May.
    22. Oechssler, Jörg & Roider, Andreas & Schmitz, Patrick W., 2009. "Cognitive abilities and behavioral biases," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 147-152, October.
    23. Scott, W. E. & Farh, Jiing-Lih & Podsakoff, Philip M., 1988. "The effects of "intrinsic" and "extrinsic" reinforcement contingencies on task behavior," Organizational Behavior and Human Decision Processes, Elsevier, vol. 41(3), pages 405-425, June.
    24. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    25. Lukas, M. & Nöth, M., 2019. "Interest rate changes and borrower search behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 163(C), pages 172-189.
    26. Uri Gneezy & Aldo Rustichini, 2000. "Pay Enough or Don't Pay at All," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(3), pages 791-810.
    27. Brice Corgnet & Antonio M. Espin & Roberto Hernán-González, 2015. "The cognitive basis of social behavior : cognitive reflection overrides antisocial but not always prosocial motives," Post-Print hal-02311954, HAL.
    28. Ciril Bosch-Rosa & Thomas Meissner & Antoni Bosch-Domènech, 2018. "Cognitive bubbles," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 132-153, March.
    29. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    30. Corgnet, Brice & Espín, Antonio M. & Hernán-González, Roberto & Kujal, Praveen & Rassenti, Stephen, 2016. "To trust, or not to trust: Cognitive reflection in trust games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 20-27.
    31. Chen, Chia-Ching & Chiu, I-Ming & Smith, John & Yamada, Tetsuji, 2013. "Too smart to be selfish? Measures of cognitive ability, social preferences, and consistency," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 112-122.
    32. Neyse, Levent & Bosworth, Steven & Ring, Patrick & Schmidt, Ulrich, 2016. "Overconfidence, Incentives and Digit Ratio," Open Access Publications from Kiel Institute for the World Economy 130145, Kiel Institute for the World Economy (IfW Kiel).
    33. Kiss, H.J. & Rodriguez-Lara, I. & Rosa-García, A., 2016. "Think twice before running! Bank runs and cognitive abilities," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 12-19.
    34. Peterson, Robert A, 2001. "On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(3), pages 450-461, December.
    35. Armin Falk & Stephan Meier & Christian Zehnder, 2013. "Do Lab Experiments Misrepresent Social Preferences? The Case Of Self-Selected Student Samples," Journal of the European Economic Association, European Economic Association, vol. 11(4), pages 839-852, August.
    36. Hoppe, Eva I. & Kusterer, David J., 2011. "Behavioral biases and cognitive reflection," Economics Letters, Elsevier, vol. 110(2), pages 97-100, February.
    37. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    38. Filippos Exadaktylos & Antonio M. Espin & Pablo Branas-Garza, 2012. "Experimental Subjects are Not Different," Working Papers 12-11, Chapman University, Economic Science Institute.
    39. Königsheim, C. & Lukas, M. & Nöth, M., 2019. "Salience theory: Calibration and heterogeneity in probability distortion," Journal of Economic Behavior & Organization, Elsevier, vol. 157(C), pages 477-495.
    40. Riedel, James A. & Nebeker, Delbert M. & Cooper, Barrie L., 1988. "The influence of monetary incentives on goal choice, goal commitment, and task performance," Organizational Behavior and Human Decision Processes, Elsevier, vol. 42(2), pages 155-180, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Frank M. Fossen & Levent Neyse & Carsten Schröder, 2025. "Does Cognitive Reflection Relate to Preferences and Socioeconomic Outcomes?," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 3(2), pages 303-343.
    2. Taylor, Matthew P., 2020. "Heterogeneous motivation and cognitive ability in the lab," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 85(C).
    3. Allred, Sarah & Duffy, Sean & Smith, John, 2016. "Cognitive load and strategic sophistication," Journal of Economic Behavior & Organization, Elsevier, vol. 125(C), pages 162-178.
    4. Nicolas Eber & Patrick Roger & Tristan Roger, 2024. "Finance and intelligence: An overview of the literature," Journal of Economic Surveys, Wiley Blackwell, vol. 38(2), pages 503-554, April.
    5. Zonca, Joshua & Coricelli, Giorgio & Polonio, Luca, 2020. "Gaze patterns disclose the link between cognitive reflection and sophistication in strategic interaction," Judgment and Decision Making, Cambridge University Press, vol. 15(2), pages 230-245, March.
    6. Chen, Chia-Ching & Chiu, I-Ming & Smith, John & Yamada, Tetsuji, 2013. "Too smart to be selfish? Measures of cognitive ability, social preferences, and consistency," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 112-122.
    7. Brañas-Garza, Pablo & Smith, John, 2016. "Cognitive abilities and economic behavior," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 1-4.
    8. Duffy, Sean & Smith, John, 2014. "Cognitive load in the multi-player prisoner's dilemma game: Are there brains in games?," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 51(C), pages 47-56.
    9. Joshua Zonca & Giorgio Coricelli & Luca Polonio, 2020. "Gaze patterns disclose the link between cognitive reflection and sophistication in strategic interaction," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(2), pages 230-245, March.
    10. Cueva, Carlos & Iturbe-Ormaetxe, Iñigo & Mata-Pérez, Esther & Ponti, Giovanni & Sartarelli, Marcello & Yu, Haihan & Zhukova, Vita, 2016. "Cognitive (ir)reflection: New experimental evidence," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 81-93.
    11. Kiss, H.J. & Rodriguez-Lara, I. & Rosa-García, A., 2016. "Think twice before running! Bank runs and cognitive abilities," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 12-19.
    12. Lau Lilleholt, 2019. "Cognitive ability and risk aversion: A systematic review and meta analysis," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(3), pages 234-279, May.
    13. Ruffle, Bradley J. & Wilson, Anne E., 2019. "Tat will tell: Tattoos and time preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 166(C), pages 566-585.
    14. Nobuyuki Hanaki, 2020. "Cognitive ability and observed behavior in laboratory experiments: implications for macroeconomic theory," The Japanese Economic Review, Springer, vol. 71(3), pages 355-378, July.
    15. Jimenez, Natalia & Rodriguez-Lara, Ismael & Tyran, Jean-Robert & Wengström, Erik, 2018. "Thinking fast, thinking badly," Economics Letters, Elsevier, vol. 162(C), pages 41-44.
    16. Corgnet, Brice & Espín, Antonio M. & Hernán-González, Roberto & Kujal, Praveen & Rassenti, Stephen, 2016. "To trust, or not to trust: Cognitive reflection in trust games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 20-27.
    17. Ludwig, Jonas & Achtziger, Anja, 2021. "Cognitive misers on the web: An online-experiment of incentives, cheating, and cognitive reflection," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 94(C).
    18. Antonio M. Espin & Valerio Capraro & Brice Corgnet & Simon Gachter & Roberto Hernan-Gonzalez & Praveen Kujal & Stephen Rassenti, 2021. "Differences in Cognitive Reflection Mediate Gender Differences in Social Preferences," Working Papers 21-22, Chapman University, Economic Science Institute.
    19. Brice Corgnet & Mark DeSantis & David Porter, 2020. "Information Aggregation and the Cognitive Make-up of Traders," Working Papers 20-18, Chapman University, Economic Science Institute.
    20. Oren Bar-Gill & Christoph Engel, 2020. "Property is Dummy Proof: An Experiment," Discussion Paper Series of the Max Planck Institute for Behavioral Economics 2020_02, Max Planck Institute for Behavioral Economics.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:soceco:v:82:y:2019:i:c:s2214804319301569. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/620175 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.