IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2402.00787.html
   My bibliography  Save this paper

Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning

Author

Listed:
  • Benjamin Patrick Evans
  • Sumitra Ganesh

Abstract

Agent-based models (ABMs) have shown promise for modelling various real world phenomena incompatible with traditional equilibrium analysis. However, a critical concern is the manual definition of behavioural rules in ABMs. Recent developments in multi-agent reinforcement learning (MARL) offer a way to address this issue from an optimisation perspective, where agents strive to maximise their utility, eliminating the need for manual rule specification. This learning-focused approach aligns with established economic and financial models through the use of rational utility-maximising agents. However, this representation departs from the fundamental motivation for ABMs: that realistic dynamics emerging from bounded rationality and agent heterogeneity can be modelled. To resolve this apparent disparity between the two approaches, we propose a novel technique for representing heterogeneous processing-constrained agents within a MARL framework. The proposed approach treats agents as constrained optimisers with varying degrees of strategic skills, permitting departure from strict utility maximisation. Behaviour is learnt through repeated simulations with policy gradients to adjust action likelihoods. To allow efficient computation, we use parameterised shared policy learning with distributions of agent skill levels. Shared policy learning avoids the need for agents to learn individual policies yet still enables a spectrum of bounded rational behaviours. We validate our model's effectiveness using real-world data on a range of canonical $n$-agent settings, demonstrating significantly improved predictive capability.

Suggested Citation

  • Benjamin Patrick Evans & Sumitra Ganesh, 2024. "Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning," Papers 2402.00787, arXiv.org.
  • Handle: RePEc:arx:papers:2402.00787
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2402.00787
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Molood Ale Ebrahim Dehkordi & Jonas Lechner & Amineh Ghorbani & Igor Nikolic & Emile Chappin & Paulien Herder, 2023. "Using Machine Learning for Agent Specifications in Agent-Based Models and Simulations: A Critical Review and Guidelines," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 26(1), pages 1-9.
    2. Hommes, Cars & Sonnemans, Joep & Tuinstra, Jan & Van De Velden, Henk, 2007. "Learning In Cobweb Experiments," Macroeconomic Dynamics, Cambridge University Press, vol. 11(S1), pages 8-33, November.
    3. repec:feb:framed:0077 is not listed on IDEAS
    4. Benjamin Patrick Evans & Mikhail Prokopenko, 2021. "Bounded rationality for relaxing best response and mutual consistency: The Quantal Hierarchy model of decision-making," Papers 2106.15844, arXiv.org, revised Mar 2023.
    5. McKelvey, Richard D. & Palfrey, Thomas R. & Weber, Roberto A., 2000. "The effects of payoff magnitude and heterogeneity on behavior in 2 x 2 games with unique mixed strategy equilibria," Journal of Economic Behavior & Organization, Elsevier, vol. 42(4), pages 523-548, August.
    6. Andrew Caplin & Mark Dean & Daniel Martin, 2011. "Search and Satisficing," American Economic Review, American Economic Association, vol. 101(7), pages 2899-2922, December.
    7. Scharfenaker, Ellis, 2020. "Implications of quantal response statistical equilibrium," Journal of Economic Dynamics and Control, Elsevier, vol. 119(C).
    8. Chong, Juin-Kuan & Ho, Teck-Hua & Camerer, Colin, 2016. "A generalized cognitive hierarchy model of games," Games and Economic Behavior, Elsevier, vol. 99(C), pages 257-274.
    9. Rogers, Brian W. & Palfrey, Thomas R. & Camerer, Colin F., 2009. "Heterogeneous quantal response equilibrium and cognitive hierarchies," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1440-1467, July.
    10. repec:feb:natura:0056 is not listed on IDEAS
    11. Russell, Golman, 2011. "Quantal response equilibria with heterogeneous agents," Journal of Economic Theory, Elsevier, vol. 146(5), pages 2013-2028, September.
    12. Richard Mckelvey & Thomas Palfrey, 1998. "Quantal Response Equilibria for Extensive Form Games," Experimental Economics, Springer;Economic Science Association, vol. 1(1), pages 9-41, June.
    13. Jean-Philippe Bouchaud, 2008. "Economics needs a scientific revolution," Nature, Nature, vol. 455(7217), pages 1181-1181, October.
    14. McKelvey Richard D. & Palfrey Thomas R., 1995. "Quantal Response Equilibria for Normal Form Games," Games and Economic Behavior, Elsevier, vol. 10(1), pages 6-38, July.
    15. Yefen Chen & Xuanming Su & Xiaobo Zhao, 2012. "Modeling Bounded Rationality in Capacity Allocation Games with the Quantal Response Equilibrium," Management Science, INFORMS, vol. 58(10), pages 1952-1962, October.
    16. Teck-Hua Ho & So-Eun Park & Xuanming Su, 2021. "A Bayesian Level- k Model in n -Person Games," Management Science, INFORMS, vol. 67(3), pages 1622-1638, March.
    17. Penghang Liu & Kshama Dwarakanath & Svitlana S Vyetrenko & Tucker Balch, 2022. "Limited or Biased: Modeling Sub-Rational Human Investors in Financial Markets," Papers 2210.08569, arXiv.org, revised Mar 2024.
    18. Sims, Christopher A., 2003. "Implications of rational inattention," Journal of Monetary Economics, Elsevier, vol. 50(3), pages 665-690, April.
    19. Steven Levitt & John List, 2008. "Homo economicus evolves," Artefactual Field Experiments 00095, The Field Experiments Website.
    20. Farmer, J. Doyne & Axtell, Robert L., 2022. "Agent-Based Modeling in Economics and Finance: Past, Present, and Future," INET Oxford Working Papers 2022-10, Institute for New Economic Thinking at the Oxford Martin School, University of Oxford.
    21. Goeree, Jacob K. & Holt, Charles A., 2004. "A model of noisy introspection," Games and Economic Behavior, Elsevier, vol. 46(2), pages 365-382, February.
    22. repec:feb:artefa:0095 is not listed on IDEAS
    23. Chen, Tao, 2018. "Round-number biases and informed trading in global markets," Journal of Business Research, Elsevier, vol. 92(C), pages 105-117.
    24. Turrell, Arthur, 2016. "Agent-based models: understanding the economy from the bottom up," Bank of England Quarterly Bulletin, Bank of England, vol. 56(4), pages 173-188.
    25. Benjamin Patrick Evans & Mikhail Prokopenko, 2021. "A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback," Papers 2102.09180, arXiv.org, revised May 2021.
    26. Benjamin Enke & Uri Gneezy & Brian Hall & David Martin & Vadim Nelidov & Theo Offerman & Jeroen van de Ven, 2023. "Cognitive Biases: Mistakes or Missing Stakes?," The Review of Economics and Statistics, MIT Press, vol. 105(4), pages 818-832, July.
    27. Callum Rhys Tilbury, 2022. "Reinforcement Learning for Economic Policy: A New Frontier?," Papers 2206.08781, arXiv.org, revised Feb 2023.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Benjamin Patrick Evans & Mikhail Prokopenko, 2021. "Bounded rationality for relaxing best response and mutual consistency: The Quantal Hierarchy model of decision-making," Papers 2106.15844, arXiv.org, revised Mar 2023.
    2. Benjamin Patrick Evans & Mikhail Prokopenko, 2024. "Bounded rationality for relaxing best response and mutual consistency: the quantal hierarchy model of decision making," Theory and Decision, Springer, vol. 96(1), pages 71-111, February.
    3. Yves Breitmoser, 2021. "Controlling for presentation effects in choice," Quantitative Economics, Econometric Society, vol. 12(1), pages 251-281, January.
    4. Golman, Russell, 2012. "Homogeneity bias in models of discrete choice with bounded rationality," Journal of Economic Behavior & Organization, Elsevier, vol. 82(1), pages 1-11.
    5. Alan Kirman & François Laisney & Paul Pezanis-Christou, 2023. "Relaxing the symmetry assumption in participation games: a specification test for cluster-heterogeneity," Experimental Economics, Springer;Economic Science Association, vol. 26(4), pages 850-878, September.
    6. Feri, Francesco & Meléndez-Jiménez, Miguel A. & Ponti, Giovanni & Vega-Redondo, Fernando, 2011. "Error cascades in observational learning: An experiment on the Chinos game," Games and Economic Behavior, Elsevier, vol. 73(1), pages 136-146, September.
    7. Choo, Lawrence C.Y & Kaplan, Todd R., 2014. "Explaining Behavior in the "11-20" Game," MPRA Paper 52808, University Library of Munich, Germany.
    8. Jacob K. Goeree & Charles A. Holt, 2001. "Ten Little Treasures of Game Theory and Ten Intuitive Contradictions," American Economic Review, American Economic Association, vol. 91(5), pages 1402-1422, December.
    9. Li, Yan & Chen, Yefen & Shou, Biying & Zhao, Xiaobo, 2019. "Oligopolistic quantity competition with bounded rationality and social comparison," International Journal of Production Economics, Elsevier, vol. 211(C), pages 180-196.
    10. Breitmoser, Yves & Tan, Jonathan H.W. & Zizzo, Daniel John, 2014. "On the beliefs off the path: Equilibrium refinement due to quantal response and level-k," Games and Economic Behavior, Elsevier, vol. 86(C), pages 102-125.
    11. Rogers, Brian W. & Palfrey, Thomas R. & Camerer, Colin F., 2009. "Heterogeneous quantal response equilibrium and cognitive hierarchies," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1440-1467, July.
    12. Aguirregabiria Victor & Xie Erhao, 2021. "Identification of Non-Equilibrium Beliefs in Games of Incomplete Information Using Experimental Data," Journal of Econometric Methods, De Gruyter, vol. 10(1), pages 1-26, January.
    13. Friedman, Evan, 2020. "Endogenous quantal response equilibrium," Games and Economic Behavior, Elsevier, vol. 124(C), pages 620-643.
    14. Goeree, Jacob K. & Holt, Charles A. & Palfrey, Thomas R., 2002. "Quantal Response Equilibrium and Overbidding in Private-Value Auctions," Journal of Economic Theory, Elsevier, vol. 104(1), pages 247-272, May.
    15. Nieken, Petra & Schmitz, Patrick W., 2012. "Repeated moral hazard and contracts with memory: A laboratory experiment," Games and Economic Behavior, Elsevier, vol. 75(2), pages 1000-1008.
    16. Carrillo, Juan D. & Palfrey, Thomas R., 2011. "No trade," Games and Economic Behavior, Elsevier, vol. 71(1), pages 66-87, January.
    17. Quement, Mark T. Le & Marcin, Isabel, 2020. "Communication and voting in heterogeneous committees: An experimental study," Journal of Economic Behavior & Organization, Elsevier, vol. 174(C), pages 449-468.
    18. Scharfenaker, Ellis, 2020. "Implications of quantal response statistical equilibrium," Journal of Economic Dynamics and Control, Elsevier, vol. 119(C).
    19. Philip A. Haile & Ali Hortaçsu & Grigory Kosenok, 2008. "On the Empirical Content of Quantal Response Equilibrium," American Economic Review, American Economic Association, vol. 98(1), pages 180-200, March.
    20. Alcocer, Christian Diego & Jeitschko, Thomas D. & Shupp, Robert, 2020. "Naive and sophisticated mixing: Experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 170(C), pages 157-173.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2402.00787. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.