IDEAS home Printed from https://ideas.repec.org/p/chu/wpaper/15-25.html
   My bibliography  Save this paper

Cognitive Reflection Test: Whom, how, when

Author

Listed:
  • Pablo Brañas-Garza

    () (Middlesex University London)

  • Praveen Kujal

    (Middlesex University London)

  • Balint Lenkei

    (Middlesex University London)

Abstract

We report the results of a meta-study of 118 Cognitive Reflection Test studies comprising of 44,558 participants across 21 countries. There is a negative correlation between being female and the overall, and individual, correct answers to CRT questions. Taking the test at the end of an experiment negatively impacts performance. Monetary incentives do not impact performance. Overall students perform better compared to non-student samples. Exposure to CRT over the years may impact outcomes, however, the effect is driven by online studies. We obtain mixed evidence on whether the sequence of questions matters. Finally, we find that computerized tests marginally improve results.

Suggested Citation

  • Pablo Brañas-Garza & Praveen Kujal & Balint Lenkei, 2015. "Cognitive Reflection Test: Whom, how, when," Working Papers 15-25, Chapman University, Economic Science Institute.
  • Handle: RePEc:chu:wpaper:15-25
    as

    Download full text from publisher

    File URL: http://www.chapman.edu/research-and-institutions/economic-science-institute/_files/WorkingPapers/cognitive-reflection-test-whom-how-when-2015.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Chen, Chia-Ching & Chiu, I-Ming & Smith, John & Yamada, Tetsuji, 2013. "Too smart to be selfish? Measures of cognitive ability, social preferences, and consistency," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 112-122.
    2. Ruffle, Bradley J. & Wilson, Anne E., 2018. "The truth about tattoos," Economics Letters, Elsevier, vol. 172(C), pages 143-147.
    3. Michael N. Stagnaro & Gordon Pennycook & David G. Rand, 2018. "Performance on the Cognitive Reflection Test is stable across time," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(3), pages 260-267, May.
    4. Oechssler, Jörg & Roider, Andreas & Schmitz, Patrick W., 2009. "Cognitive abilities and behavioral biases," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 147-152, October.
    5. Scott, W. E. & Farh, Jiing-Lih & Podsakoff, Philip M., 1988. "The effects of "intrinsic" and "extrinsic" reinforcement contingencies on task behavior," Organizational Behavior and Human Decision Processes, Elsevier, vol. 41(3), pages 405-425, June.
    6. Nobuyuki Hanaki & Nicolas Jacquemet & Stéphane Luchini & Adam Zylbersztejn, 2016. "Fluid intelligence and cognitive reflection in a strategic environment: evidence from dominance-solvable games," Post-Print hal-01359231, HAL.
    7. Ciril Bosch-Rosa & Thomas Meissner & Antoni Bosch-Domènech, 2018. "Cognitive bubbles," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 132-153, March.
    8. Andersson, Ola & Tyran, Jean-Robert & Wengström, Erik & Holm, Håkan J., 2013. "Risk Aversion Relates to Cognitive Ability: Fact or Fiction?," Working Paper Series 964, Research Institute of Industrial Economics.
    9. Lukas, M. & Nöth, M., 2019. "Interest rate changes and borrower search behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 163(C), pages 172-189.
    10. Armin Falk & James J. Heckman, 2009. "Lab Experiments are a Major Source of Knowledge in the Social Sciences," Working Papers 200935, Geary Institute, University College Dublin.
    11. Brañas-Garza, Pablo & García-Muñoz, Teresa & González, Roberto Hernán, 2012. "Cognitive effort in the Beauty Contest Game," Journal of Economic Behavior & Organization, Elsevier, vol. 83(2), pages 254-260.
    12. Uri Gneezy & Aldo Rustichini, 2000. "Pay Enough or Don't Pay at All," The Quarterly Journal of Economics, Oxford University Press, vol. 115(3), pages 791-810.
    13. Brice Corgnet & Antonio M. Espin & Roberto Hernán-González, 2015. "The cognitive basis of social behavior : cognitive reflection overrides antisocial but not always prosocial motives," Post-Print hal-02311954, HAL.
    14. Besedeš, Tibor & Deck, Cary & Sarangi, Sudipta & Shor, Mikhael, 2012. "Decision-making strategies and performance among seniors," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 524-533.
    15. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    16. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    17. Ola Andersson & Håkan J. Holm & Jean-Robert Tyran & Erik Wengström, 2016. "Risk Aversion Relates to Cognitive Ability: Preferences Or Noise?," Journal of the European Economic Association, European Economic Association, vol. 14(5), pages 1129-1154.
    18. Brice Corgnet & Roberto Hernán-González & Praveen Kujal & David Porter, 2015. "The Effect of Earned Versus House Money on Price Bubble Formation in Experimental Asset Markets," Review of Finance, European Finance Association, vol. 19(4), pages 1455-1488.
    19. Hoppe, Eva I. & Kusterer, David J., 2011. "Behavioral biases and cognitive reflection," Economics Letters, Elsevier, vol. 110(2), pages 97-100, February.
    20. Carpenter, Jeffrey & Graham, Michael & Wolf, Jesse, 2013. "Cognitive ability and strategic sophistication," Games and Economic Behavior, Elsevier, vol. 80(C), pages 115-130.
    21. Corgnet, Brice & Espín, Antonio M. & Hernán-González, Roberto & Kujal, Praveen & Rassenti, Stephen, 2016. "To trust, or not to trust: Cognitive reflection in trust games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 20-27.
    22. Neyse, Levent & Bosworth, Steven & Ring, Patrick & Schmidt, Ulrich, 2016. "Overconfidence, Incentives and Digit Ratio," Open Access Publications from Kiel Institute for the World Economy 130145, Kiel Institute for the World Economy (IfW).
    23. Bonner, Sarah E. & Sprinkle, Geoffrey B., 2002. "The effects of monetary incentives on effort and task performance: theories, evidence, and a framework for research," Accounting, Organizations and Society, Elsevier, vol. 27(4-5), pages 303-345.
    24. Douglas W. Diamond & Philip H. Dybvig, 2000. "Bank runs, deposit insurance, and liquidity," Quarterly Review, Federal Reserve Bank of Minneapolis, vol. 24(Win), pages 14-23.
    25. Cheung, Stephen L. & Hedegaard, Morten & Palan, Stefan, 2014. "To see is to believe: Common expectations in experimental asset markets," European Economic Review, Elsevier, vol. 66(C), pages 84-96.
    26. Cueva, Carlos & Iturbe-Ormaetxe, Iñigo & Mata-Pérez, Esther & Ponti, Giovanni & Sartarelli, Marcello & Yu, Haihan & Zhukova, Vita, 2016. "Cognitive (ir)reflection: New experimental evidence," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 81-93.
    27. Charles N. Noussair & Stefan T. Trautmann & Gijs van de Kuilen, 2014. "Higher Order Risk Attitudes, Demographics, and Financial Decisions," Review of Economic Studies, Oxford University Press, vol. 81(1), pages 325-355.
    28. Kiss, H.J. & Rodriguez-Lara, I. & Rosa-García, A., 2016. "Think twice before running! Bank runs and cognitive abilities," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 12-19.
    29. Ben-Ner, Avner & Kong, Fanmin & Putterman, Louis, 2004. "Share and share alike? Gender-pairing, personality, and cognitive ability as determinants of giving," Journal of Economic Psychology, Elsevier, vol. 25(5), pages 581-589, October.
    30. Guillermo Campitelli & Martin Labollita, 2010. "Correlations of cognitive reflection with judgments and choices," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(3), pages 182-191, June.
    31. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    32. Peterson, Robert A, 2001. "On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis," Journal of Consumer Research, Oxford University Press, vol. 28(3), pages 450-461, December.
    33. Armin Falk & Stephan Meier & Christian Zehnder, 2013. "Do Lab Experiments Misrepresent Social Preferences? The Case Of Self-Selected Student Samples," Journal of the European Economic Association, European Economic Association, vol. 11(4), pages 839-852, August.
    34. Steven D. Levitt & John A. List, 2007. "What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?," Journal of Economic Perspectives, American Economic Association, vol. 21(2), pages 153-174, Spring.
    35. Filippos Exadaktylos & Antonio M. Espin & Pablo Branas-Garza, 2012. "Experimental Subjects are Not Different," Working Papers 12-11, Chapman University, Economic Science Institute.
    36. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    37. Königsheim, C. & Lukas, M. & Nöth, M., 2019. "Salience theory: Calibration and heterogeneity in probability distortion," Journal of Economic Behavior & Organization, Elsevier, vol. 157(C), pages 477-495.
    38. Riedel, James A. & Nebeker, Delbert M. & Cooper, Barrie L., 1988. "The influence of monetary incentives on goal choice, goal commitment, and task performance," Organizational Behavior and Human Decision Processes, Elsevier, vol. 42(2), pages 155-180, October.
    39. Keela S. Thomson & Daniel M. Oppenheimer, 2016. "Investigating an alternate form of the cognitive reflection test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(1), pages 99-113, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin Enke & Uri Gneezy & Brian Hall & David Martin & Vadim Nelidov & Theo Offerman & Jeroen van de Ven, 2020. "Cognitive Biases: Mistakes or Missing Stakes?," CESifo Working Paper Series 8168, CESifo.
    2. Holt, Charles A. & Porzio, Megan & Song, Michelle Yingze, 2017. "Price bubbles, gender, and expectations in experimental asset markets," European Economic Review, Elsevier, vol. 100(C), pages 72-94.
    3. Brice Corgnet & Mark Desantis & David Porter, 2018. "What Makes a Good Trader? On the Role of Intuition and Reflection on Trader Performance," Journal of Finance, American Finance Association, vol. 73(3), pages 1113-1137, June.
    4. Judit Alonso & Roberto Di Paolo & Giovanni Ponti & Marcello Sartarelli, 2017. "Some (Mis)facts about 2D:4D, Preferences and Personality," Working Papers. Serie AD 2017-08, Instituto Valenciano de Investigaciones Económicas, S.A. (Ivie).
    5. Noussair, Charles N. & Tucker, Steven & Xu, Yilong, 2016. "Futures markets, cognitive ability, and mispricing in experimental asset markets," Journal of Economic Behavior & Organization, Elsevier, vol. 130(C), pages 166-179.
    6. Valerio Capraro & Brice Corgnet & Antonio M. Espin & Roberto Hernán-Gonzalez, 2017. "Deliberation favours social efficiency by making people disregard their relative shares: Evidence from US and India," Post-Print halshs-01439121, HAL.
    7. Krawczyk, Michał & Sylwestrzak, Marta, 2018. "Exploring the role of deliberation time in non-selfish behavior: The double response method," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 72(C), pages 121-134.
    8. Jimenez, Natalia & Rodriguez-Lara, Ismael & Tyran, Jean-Robert & Wengström, Erik, 2018. "Thinking fast, thinking badly," Economics Letters, Elsevier, vol. 162(C), pages 41-44.
    9. Amador, Luis & Brañas-Garza, Pablo & Espín, Antonio M. & Garcia, Teresa & Hernández, Ana, 2019. "Consistent and inconsistent choices under uncertainty: The role of cognitive abilities," MPRA Paper 95178, University Library of Munich, Germany.
    10. Muñoz-Murillo, Melisa & Álvarez-Franco, Pilar B. & Restrepo-Tobón, Diego A., 2020. "The role of cognitive abilities on financial literacy: New experimental evidence," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 84(C).
    11. Ruffle, Bradley J. & Wilson, Anne E., 2019. "Tat will tell: Tattoos and time preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 166(C), pages 566-585.
    12. Jordi Brandts & Isabel Busom & Cristina Lopez-Mayan & Judith Panadés, 2019. "Dispelling Misconceived Beliefs: Insights from Experiments," Working Papers 1096, Barcelona Graduate School of Economics.
    13. Peter Hoeschler & Uschi Backes-Gellner, 2017. "The Relative Importance of Personal Characteristics for the Hiring of Young Workers," Economics of Education Working Paper Series 0142, University of Zurich, Department of Business Administration (IBW), revised Jan 2018.
    14. Andrew Meyer & Elizabeth Zhou & Shane Frederick, 2018. "The non-effects of repeated exposure to the Cognitive Reflection Test," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(3), pages 246-259, May.
    15. Nobuyuki Hanaki & Nicolas Jacquemet & Stéphane Luchini & Adam Zylbersztejn, 2016. "Fluid intelligence and cognitive reflection in a strategic environment: evidence from dominance-solvable games," Post-Print hal-01359231, HAL.
    16. Marcello Sartarelli, 2016. "Handedness, Ability, Earnings and Risk. Evidence from the Lab," Working Papers. Serie AD 2016-04, Instituto Valenciano de Investigaciones Económicas, S.A. (Ivie).
    17. Brown, Alexander L. & Viriyavipart, Ajalavat & Wang, Xiaoyuan, 2018. "Search deterrence in experimental consumer goods markets," European Economic Review, Elsevier, vol. 104(C), pages 167-184.
    18. Alós-Ferrer, Carlos & Hügelschäfer, Sabine, 2016. "Faith in intuition and cognitive reflection," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 61-70.
    19. Skagerlund, Kenny & Lind, Thérèse & Strömbäck, Camilla & Tinghög, Gustav & Västfjäll, Daniel, 2018. "Financial literacy and the role of numeracy–How individuals’ attitude and affinity with numbers influence financial literacy," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 74(C), pages 18-25.
    20. Avram Laura-Augustina, 2018. "Gender Differences and Other Findings on the Cognitive Reflection Test," Studia Universitatis Babeș-Bolyai Oeconomica, Sciendo, vol. 63(3), pages 56-67, December.
    21. Taylor, Matthew P., 2020. "Heterogeneous motivation and cognitive ability in the lab," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 85(C).
    22. Brañas-Garza, Pablo & Smith, John, 2016. "Cognitive abilities and economic behavior," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 1-4.

    More about this item

    Keywords

    CRT; Experiments; Gender; Incentives; Glucose and Cognition;

    JEL classification:

    • Z00 - Other Special Topics - - General - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:chu:wpaper:15-25. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Megan Luetje). General contact details of provider: http://edirc.repec.org/data/esichus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.