IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2309.05898.html

Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing

Author

Listed:
  • Nunzio Lor`e
  • Babak Heydari

Abstract

This paper investigates the strategic decision-making capabilities of three Large Language Models (LLMs): GPT-3.5, GPT-4, and LLaMa-2, within the framework of game theory. Utilizing four canonical two-player games -- Prisoner's Dilemma, Stag Hunt, Snowdrift, and Prisoner's Delight -- we explore how these models navigate social dilemmas, situations where players can either cooperate for a collective benefit or defect for individual gain. Crucially, we extend our analysis to examine the role of contextual framing, such as diplomatic relations or casual friendships, in shaping the models' decisions. Our findings reveal a complex landscape: while GPT-3.5 is highly sensitive to contextual framing, it shows limited ability to engage in abstract strategic reasoning. Both GPT-4 and LLaMa-2 adjust their strategies based on game structure and context, but LLaMa-2 exhibits a more nuanced understanding of the games' underlying mechanics. These results highlight the current limitations and varied proficiencies of LLMs in strategic decision-making, cautioning against their unqualified use in tasks requiring complex strategic reasoning.

Suggested Citation

  • Nunzio Lor`e & Babak Heydari, 2023. "Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing," Papers 2309.05898, arXiv.org.
    • Handle: RePEc:arx:papers:2309.05898
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2309.05898
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yiting Chen & Tracy Xiao Liu & You Shan & Songfa Zhong, 2023. "The emergence of economic rationality of GPT," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(51), pages 2316205120-, December.
    2. Fulin Guo, 2023. "GPT in Game Theory Experiments," Papers 2305.05516, arXiv.org, revised Dec 2023.
    3. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org, revised Feb 2026.
    4. Joseph N. Luchman, 2021. "Determining relative importance in Stata using dominance analysis: domin and domme," Stata Journal, StataCorp LLC, vol. 21(2), pages 510-538, June.
    5. Steve Phelps & Yvan I. Russell, 2023. "The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?," Papers 2305.07970, arXiv.org, revised Jun 2024.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cagno, Daniela Di & Lin, Lihui, 2025. "How do individuals interact with an AI advisor in strategic reasoning? An experimental study in beauty contest," Journal of Economic Behavior & Organization, Elsevier, vol. 237(C).
    2. Ayato Kitadai & Sinndy Dayana Rico Lugo & Yudai Tsurusaki & Yusuke Fukasawa & Nariaki Nishino, 2024. "Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?," Papers 2406.11426, arXiv.org.
    3. Chuanhao Li & Runhan Yang & Tiankai Li & Milad Bafarassat & Kourosh Sharifi & Dirk Bergemann & Zhuoran Yang, 2024. "STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making," Cowles Foundation Discussion Papers 2393, Cowles Foundation for Research in Economics, Yale University.
    4. Eléonore Dodivers & Ismaël Rafaï, 2025. "Uncovering the Fairness of AI: Exploring Focal Point, Inequality Aversion, and Altruism in ChatGPT's Dictator Game Decisions," GREDEG Working Papers 2025-09, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    5. Jin Han & Balaraju Battu & Ivan Romić & Talal Rahwan & Petter Holme, 2025. "Static network structure cannot stabilize cooperation among large language model agents," PLOS ONE, Public Library of Science, vol. 20(5), pages 1-16, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kirshner, Samuel N., 2024. "GPT and CLT: The impact of ChatGPT's level of abstraction on consumer recommendations," Journal of Retailing and Consumer Services, Elsevier, vol. 76(C).
    2. Christoph Engel & Max R. P. Grossmann & Axel Ockenfels, 2023. "Integrating machine behavior into human subject experiments: A user-friendly toolkit and illustrations," Discussion Paper Series of the Max Planck Institute for Behavioral Economics 2024_01, Max Planck Institute for Behavioral Economics.
    3. Bauer, Kevin & Liebich, Lena & Hinz, Oliver & Kosfeld, Michael, 2023. "Decoding GPT's hidden "rationality" of cooperation," SAFE Working Paper Series 401, Leibniz Institute for Financial Research SAFE.
    4. Philip Brookins & Jason DeBacker, 2024. "Playing games with GPT: What can we learn about a large language model from canonical strategic games?," Economics Bulletin, AccessEcon, vol. 44(1), pages 25-37.
    5. Jiafu An & Difang Huang & Chen Lin & Mingzhu Tai, 2024. "Measuring Gender and Racial Biases in Large Language Models," Papers 2403.15281, arXiv.org.
    6. Jingru Jia & Zehua Yuan & Junhao Pan & Paul E. McNamara & Deming Chen, 2024. "Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context," Papers 2406.05972, arXiv.org, revised Oct 2024.
    7. Thomas R. Cook & Sophia Kazinnik & Zach Modig & Nathan M. Palmer, 2025. "What Do LLMs Want?," Research Working Paper RWP 25-19, Federal Reserve Bank of Kansas City.
    8. Cagno, Daniela Di & Lin, Lihui, 2025. "How do individuals interact with an AI advisor in strategic reasoning? An experimental study in beauty contest," Journal of Economic Behavior & Organization, Elsevier, vol. 237(C).
    9. Ennio Bilancini & Leonardo Boncinelli & Eugenio Vicario, 2024. "AI-powered Chatbots: Effective Communication Styles for Sustainable Development Goals," Papers 2407.01057, arXiv.org.
    10. Yingnan Yan & Tianming Liu & Yafeng Yin, 2025. "Valuing Time in Silicon: Can Large Language Models Replicate Human Value of Travel Time," Papers 2507.22244, arXiv.org, revised Dec 2025.
    11. Ali Goli & Amandeep Singh, 2024. "Frontiers: Can Large Language Models Capture Human Preferences?," Marketing Science, INFORMS, vol. 43(4), pages 709-722, July.
    12. Iñaki Aldasoro & Ajit Desai, 2025. "Money Talks: AI Agents for Cash Management in Payment Systems," Staff Working Papers 25-35, Bank of Canada.
    13. Vítor Castro & Rodrigo Martins, 2024. "Lockdowns, vaccines, and the economy: How economic perceptions were shaped during the COVID‐19 pandemic†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 71(3), pages 439-456, July.
    14. Kevin Leyton-Brown & Paul Milgrom & Neil Newman & Ilya Segal, 2024. "Artificial Intelligence and Market Design: Lessons Learned from Radio Spectrum Reallocation," NBER Chapters, in: New Directions in Market Design, pages 119-151, National Bureau of Economic Research, Inc.
    15. Pawe{l} Niszczota & Tomasz Grzegorczyk & Alexander Pastukhov, 2025. "People Are Highly Cooperative with Large Language Models, Especially When Communication Is Possible or Following Human Interaction," Papers 2507.18639, arXiv.org.
    16. C. Monica Capra & Thomas J. Kniesner, 2025. "Daniel Kahneman’s underappreciated last published paper: Empirical implications for benefit-cost analysis and a chat session discussion with bots," Journal of Risk and Uncertainty, Springer, vol. 71(1), pages 29-51, August.
    17. Jolene Tan, 2023. "Perceptions towards pronatalist policies in Singapore," Journal of Population Research, Springer, vol. 40(3), pages 1-27, September.
    18. Hongshen Sun & Juanjuan Zhang, 2025. "From Model Choice to Model Belief: Establishing a New Measure for LLM-Based Research," Papers 2512.23184, arXiv.org.
    19. Shu Wang & Zijun Yao & Shuhuai Zhang & Jianuo Gai & Tracy Xiao Liu & Songfa Zhong, 2025. "When Experimental Economics Meets Large Language Models: Evidence-based Tactics," Papers 2505.21371, arXiv.org, revised Jul 2025.
    20. Iñaki Aldasoro & Ajit Desai, 2025. "AI agents for cash management in payment systems," BIS Working Papers 1310, Bank for International Settlements.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2309.05898. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.