IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2402.09552.html
   My bibliography  Save this paper

STEER: Assessing the Economic Rationality of Large Language Models

Author

Listed:
  • Narun Raman
  • Taylor Lundy
  • Samuel Amouyal
  • Yoav Levine
  • Kevin Leyton-Brown
  • Moshe Tennenholtz

Abstract

There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions -- and more broadly, determining whether an LLM agent is reliable enough to be trusted -- requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "STEER report card." Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.

Suggested Citation

  • Narun Raman & Taylor Lundy & Samuel Amouyal & Yoav Levine & Kevin Leyton-Brown & Moshe Tennenholtz, 2024. "STEER: Assessing the Economic Rationality of Large Language Models," Papers 2402.09552, arXiv.org, revised May 2024.
  • Handle: RePEc:arx:papers:2402.09552
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2402.09552
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kreps, David M & Wilson, Robert, 1982. "Sequential Equilibria," Econometrica, Econometric Society, vol. 50(4), pages 863-894, July.
    2. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    3. Parayre, Roch, 1995. "The strategic implications of sunk costs: A behavioral perspective," Journal of Economic Behavior & Organization, Elsevier, vol. 28(3), pages 417-442, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Ludwig & Alexander Zimper, 2013. "A decision-theoretic model of asset-price underreaction and overreaction to dividend news," Annals of Finance, Springer, vol. 9(4), pages 625-665, November.
    2. Daniel Friedman & Kai Pommerenke & Rajan Lukose & Garrett Milam & Bernardo Huberman, 2007. "Searching for the sunk cost fallacy," Experimental Economics, Springer;Economic Science Association, vol. 10(1), pages 79-104, March.
    3. Sendhil Mullainathan & Joshua Schwartzstein & Andrei Shleifer, 2008. "Coarse Thinking and Persuasion," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(2), pages 577-619.
    4. Kjell Hausken, 1997. "Game-theoretic and Behavioral Negotiation Theory," Group Decision and Negotiation, Springer, vol. 6(6), pages 511-528, December.
    5. Robin Chark & Vincent Mak & A. V. Muthukrishnan, 2020. "The premium as informational cue in insurance decision making," Theory and Decision, Springer, vol. 88(3), pages 369-404, April.
    6. Ludwig, Alexander & Zimper, Alexander, 2006. "Investment behavior under ambiguity: The case of pessimistic decision makers," Mathematical Social Sciences, Elsevier, vol. 52(2), pages 111-130, September.
    7. van Damme, E.E.C., 2015. "Game theory : Noncooperative games," Other publications TiSEM ff518f2b-501f-4d99-817b-c, Tilburg University, School of Economics and Management.
    8. Grillo, Edoardo, 2016. "The hidden cost of raising voters’ expectations: Reference dependence and politicians’ credibility," Journal of Economic Behavior & Organization, Elsevier, vol. 130(C), pages 126-143.
    9. Seow Eng Ong & Davin Wang & Calvin Chua, 2023. "Disruptive Innovation and Real Estate Agency: The Disruptee Strikes Back," The Journal of Real Estate Finance and Economics, Springer, vol. 67(2), pages 287-317, August.
    10. Herrmann, Tabea & Hübler, Olaf & Menkhoff, Lukas & Schmidt, Ulrich, 2016. "Allais for the poor," Kiel Working Papers 2036, Kiel Institute for the World Economy (IfW Kiel).
    11. Christiane Goodfellow & Dirk Schiereck & Steffen Wippler, 2013. "Are behavioural finance equity funds a superior investment? A note on fund performance and market efficiency," Journal of Asset Management, Palgrave Macmillan, vol. 14(2), pages 111-119, April.
    12. Berg, Joyce E. & Rietz, Thomas A., 2019. "Longshots, overconfidence and efficiency on the Iowa Electronic Market," International Journal of Forecasting, Elsevier, vol. 35(1), pages 271-287.
    13. Reckers, Philip M.J. & Sanders, Debra L. & Roark, Stephen J., 1994. "The Influence of Ethical Attitudes on Taxpayer Compliance," National Tax Journal, National Tax Association;National Tax Journal, vol. 47(4), pages 825-836, December.
    14. Bier, Vicki & Gutfraind, Alexander, 2019. "Risk analysis beyond vulnerability and resilience – characterizing the defensibility of critical systems," European Journal of Operational Research, Elsevier, vol. 276(2), pages 626-636.
    15. Sitinjak Elizabeth Lucky Maretha & Haryanti Kristiana & Kurniasari Widuri & Sasmito Yohanes Wisnu Djati, 2019. "Investor behavior based on personality and company life cycle," HOLISTICA – Journal of Business and Public Administration, Sciendo, vol. 10(2), pages 23-38, August.
    16. Theo Arentze & Tao Feng & Harry Timmermans & Jops Robroeks, 2012. "Context-dependent influence of road attributes and pricing policies on route choice behavior of truck drivers: results of a conjoint choice experiment," Transportation, Springer, vol. 39(6), pages 1173-1188, November.
    17. van den Bergh, J.C.J.M. & Botzen, W.J.W., 2015. "Monetary valuation of the social cost of CO2 emissions: A critical survey," Ecological Economics, Elsevier, vol. 114(C), pages 33-46.
    18. Frank D. Hodge & Roger D. Martin & Jamie H. Pratt, 2006. "Audit Qualifications of Income†Decreasing Accounting Choices," Contemporary Accounting Research, John Wiley & Sons, vol. 23(2), pages 369-394, June.
    19. Philippe Fevrier & Sebastien Gay, 2005. "Informed Consent Versus Presumed Consent The Role of the Family in Organ Donations," HEW 0509007, University Library of Munich, Germany.
    20. Ran Sun Lyng & Jie Zhou, 2019. "Household Portfolio Choice Before and After a House Purchase," Economics Working Papers 2019-01, Department of Economics and Business Economics, Aarhus University.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2402.09552. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.