IDEAS home Printed from https://ideas.repec.org/a/eee/jeborg/v237y2025ics0167268125002781.html

How do individuals interact with an AI advisor in strategic reasoning? An experimental study in beauty contest

Author

Listed:
  • Cagno, Daniela Di
  • Lin, Lihui

Abstract

This paper experimentally investigates how individuals use generative AI to learn and respond in a strategic reasoning contest. An advisor based on level k theory and implemented using ChatGPT is introduced in a four-stage beauty contest experiment. The experiment is designed to explore how AI advisors influence the depth of human reasoning by shaping beliefs, learning, and sophisticated backward induction. Extended cognitive hierarchy models (Camerer et al., 2004) are applied to identify heterogeneous level distribution and more sophisticated thinking. Additionally, the interactions between participants with and without AI advisors are examined. Two key results emerge. First, individuals overestimate AI capabilities when competing against AI-guided participants, which motivates them to employ higher levels of thinking. This observed higher-level behaviour is driven by more sophisticated backward reasoning. Second, improved reasoning under AI guidance shows heterogeneous effects across Cognitive Reflection Test scores, suggesting that AI's impact depends on participants' pre-existing cognitive abilities. Overall, this early research provides insights into the interaction between generative AI and human cognition and reasoning.

Suggested Citation

  • Cagno, Daniela Di & Lin, Lihui, 2025. "How do individuals interact with an AI advisor in strategic reasoning? An experimental study in beauty contest," Journal of Economic Behavior & Organization, Elsevier, vol. 237(C).
    • Handle: RePEc:eee:jeborg:v:237:y:2025:i:c:s0167268125002781
    DOI: 10.1016/j.jebo.2025.107159
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167268125002781
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jebo.2025.107159?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Yiting Chen & Tracy Xiao Liu & You Shan & Songfa Zhong, 2023. "The emergence of economic rationality of GPT," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(51), pages 2316205120-, December.
    2. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde, 2010. "Are Risk Aversion and Impatience Related to Cognitive Ability?," American Economic Review, American Economic Association, vol. 100(3), pages 1238-1260, June.
    3. Alaoui, Larbi & Janezic, Katharina A. & Penta, Antonio, 2020. "Reasoning about others' reasoning," Journal of Economic Theory, Elsevier, vol. 189(C).
    4. Rydval, Ondrej & Ortmann, Andreas & Ostatnicky, Michal, 2009. "Three very simple games and what it takes to solve them," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 589-601, October.
    5. McCannon, Bryan C., 2024. "Artificial intelligence is a pro-social norm complier," Economics Letters, Elsevier, vol. 241(C).
    6. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    7. Ball, Sheryl & Katz, Benjamin & Li, Flora & Smith, Alec, 2023. "The effect of cognitive load on economic decision-making: a replication attempt," Journal of Economic Behavior & Organization, Elsevier, vol. 210(C), pages 226-242.
    8. Chris Forman & Avi Goldfarb & Shane Greenstein, 2012. "The Internet and Local Wages: A Puzzle," American Economic Review, American Economic Association, vol. 102(1), pages 556-575, February.
    9. David Gill & Victoria Prowse, 2016. "Cognitive Ability, Character Skills, and Learning to Play Equilibrium: A Level-k Analysis," Journal of Political Economy, University of Chicago Press, vol. 124(6), pages 1619-1676.
    10. Colin Camerer & Teck-Hua Ho, 1999. "Experience-weighted Attraction Learning in Normal Form Games," Econometrica, Econometric Society, vol. 67(4), pages 827-874, July.
    11. Ho, Teck-Hua & Camerer, Colin & Weigelt, Keith, 1998. "Iterated Dominance and Iterated Best Response in Experimental "p-Beauty Contests."," American Economic Review, American Economic Association, vol. 88(4), pages 947-969, September.
    12. Jeanine Miklós-Thal & Catherine Tucker, 2019. "Collusion by Algorithm: Does Better Demand Prediction Facilitate Coordination Between Sellers?," Management Science, INFORMS, vol. 65(4), pages 1552-1561, April.
    13. Agrawal, Ajay & Gans, Joshua S. & Goldfarb, Avi, 2019. "Exploring the impact of artificial Intelligence: Prediction versus judgment," Information Economics and Policy, Elsevier, vol. 47(C), pages 1-6.
    14. Qiaozhu Mei & Yutong Xie & Walter Yuan & Matthew O. Jackson, 2024. "A Turing test of whether AI chatbots are behaviorally similar to humans," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 121(9), pages 2313925121-, February.
    15. Daron Acemoglu & Pascual Restrepo, 2018. "The Race between Man and Machine: Implications of Technology for Growth, Factor Shares, and Employment," American Economic Review, American Economic Association, vol. 108(6), pages 1488-1542, June.
    16. Stahl, Dale O., 2000. "Rule Learning in Symmetric Normal-Form Games: Theory and Evidence," Games and Economic Behavior, Elsevier, vol. 32(1), pages 105-138, July.
    17. Burnham, Terence C. & Cesarini, David & Johannesson, Magnus & Lichtenstein, Paul & Wallace, Björn, 2009. "Higher cognitive ability is associated with lower entries in a p-beauty contest," Journal of Economic Behavior & Organization, Elsevier, vol. 72(1), pages 171-175, October.
    18. M. Kate Bundorf & Maria Polyakova & Ming Tai-Seale, 2024. "How Do Consumers Interact with Digital Expert Advice? Experimental Evidence from Health Insurance," Management Science, INFORMS, vol. 70(11), pages 7617-7643, November.
    19. Alexander Bick & Adam Blandin & David Deming, 2023. "The Rapid Adoption of Generative AI," On the Economy 98843, Federal Reserve Bank of St. Louis.
    20. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org, revised Feb 2026.
    21. Teck-Hua Ho & So-Eun Park & Xuanming Su, 2021. "A Bayesian Level- k Model in n -Person Games," Management Science, INFORMS, vol. 67(3), pages 1622-1638, March.
    22. Colin F. Camerer & Teck-Hua Ho & Juin-Kuan Chong, 2004. "A Cognitive Hierarchy Model of Games," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(3), pages 861-898.
    23. Gil Cohen, 2022. "Algorithmic Trading and Financial Forecasting Using Advanced Artificial Intelligence Methodologies," Mathematics, MDPI, vol. 10(18), pages 1-13, September.
    24. Nunzio Lor`e & Babak Heydari, 2023. "Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing," Papers 2309.05898, arXiv.org.
    25. Elizabeth Han & Dezhi Yin & Han Zhang, 2023. "Bots with Feelings: Should AI Agents Express Positive Emotion in Customer Service?," Information Systems Research, INFORMS, vol. 34(3), pages 1296-1311, September.
    26. Richard H. Thaler, 2016. "Behavioral Economics: Past, Present, and Future," American Economic Review, American Economic Association, vol. 106(7), pages 1577-1600, July.
    27. Jonas Hjort & Jonas Poulsen, 2019. "The Arrival of Fast Internet and Employment in Africa," American Economic Review, American Economic Association, vol. 109(3), pages 1032-1079, March.
    28. Callen Anthony & Beth A. Bechky & Anne-Laure Fayard, 2023. "“Collaborating” with AI: Taking a System View to Explore the Future of Work," Organization Science, INFORMS, vol. 34(5), pages 1672-1694, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Filippo Gusella & Eugenio Vicario, 2025. "Generative Agents and Expectations: Do LLMs Align with Heterogeneous Agent Models?," Working Papers - Economics wp2025_18.rdf, Universita' degli Studi di Firenze, Dipartimento di Scienze per l'Economia e l'Impresa.
    2. Filippo Gusella & Eugenio Vicario, 2025. "Generative Agents and Expectations: Do LLMs Align with Heterogeneous Agent Models?," Papers 2511.08604, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alekseenko, Iuliia & Dagaev, Dmitry & Paklina, Sofiia & Parshakov, Petr, 2025. "Strategizing with AI: Insights from a beauty contest experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 240(C).
    2. Ye Jin, 2021. "Does level-k behavior imply level-k thinking?," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 330-353, March.
    3. Joshua Zonca & Giorgio Coricelli & Luca Polonio, 2020. "Gaze patterns disclose the link between cognitive reflection and sophistication in strategic interaction," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(2), pages 230-245, March.
    4. Teck-Hua Ho & So-Eun Park & Xuanming Su, 2021. "A Bayesian Level- k Model in n -Person Games," Management Science, INFORMS, vol. 67(3), pages 1622-1638, March.
    5. Feng, Jun & Qin, Xiangdong & Wang, Xiaoyuan, 2021. "A Bayesian cognitive hierarchy model with fixed reasoning levels," Journal of Economic Behavior & Organization, Elsevier, vol. 192(C), pages 704-723.
    6. Gill, David & Prowse, Victoria, 2012. "Cognitive ability and learning to play equilibrium: A level-k analysis," MPRA Paper 38317, University Library of Munich, Germany, revised 23 Apr 2012.
    7. Zonca, Joshua & Coricelli, Giorgio & Polonio, Luca, 2020. "Gaze patterns disclose the link between cognitive reflection and sophistication in strategic interaction," Judgment and Decision Making, Cambridge University Press, vol. 15(2), pages 230-245, March.
    8. Bayer, R.-C. & Renou, Ludovic, 2016. "Logical abilities and behavior in strategic-form games," Journal of Economic Psychology, Elsevier, vol. 56(C), pages 39-59.
    9. King King Li & Kang Rong, 2024. "A two-step guessing game," Theory and Decision, Springer, vol. 97(1), pages 89-108, August.
    10. Ho, Teck H. & Camerer, Colin F. & Chong, Juin-Kuan, 2007. "Self-tuning experience weighted attraction learning in games," Journal of Economic Theory, Elsevier, vol. 133(1), pages 177-198, March.
    11. Cooper, Kristen & Schneider, Henry & Waldman, Michael, 2021. "Limited rationality and the strategic environment: Further evidence from a pricing game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 90(C).
    12. Ballester, Coralio & Rodriguez-Moral, Antonio & Vorsatz, Marc, 2024. "Cognitive reflection in experimental anchored guessing games," Games and Economic Behavior, Elsevier, vol. 148(C), pages 179-195.
    13. Shu-Heng Chen & Ye-Rong Du & Lee-Xieng Yang, 2014. "Cognitive capacity and cognitive hierarchy: a study based on beauty contest experiments," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 9(1), pages 69-105, April.
    14. Wanqun Zhao, 2020. "Cost of Reasoning and Strategic Sophistication," Games, MDPI, vol. 11(3), pages 1-27, September.
    15. Cooper, Kristen B. & Schneider, Henry S. & Waldman, Michael, 2017. "Limited rationality and the strategic environment: Further theory and experimental evidence," Games and Economic Behavior, Elsevier, vol. 106(C), pages 188-208.
    16. Lohse, Johannes & Rahal, Rima-Maria & Schulte-Mecklenbeck, Michael & Sofianos, Andis & Wollbrant, Conny, 2024. "Investigations of decision processes at the intersection of psychology and economics," Journal of Economic Psychology, Elsevier, vol. 103(C).
    17. Allred, Sarah & Duffy, Sean & Smith, John, 2016. "Cognitive load and strategic sophistication," Journal of Economic Behavior & Organization, Elsevier, vol. 125(C), pages 162-178.
    18. Lin, Po-Hsuan & Palfrey, Thomas R., 2024. "Cognitive hierarchies for games in extensive form," Journal of Economic Theory, Elsevier, vol. 220(C).
    19. Anufriev, Mikhail & Duffy, John & Panchenko, Valentyn, 2022. "Learning in two-dimensional beauty contest games: Theory and experimental evidence," Journal of Economic Theory, Elsevier, vol. 201(C).
    20. King King Li & Kang Rong, 2023. "A Two-Step Guessing Game," Post-Print hal-04376266, HAL.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jeborg:v:237:y:2025:i:c:s0167268125002781. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jebo .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.