IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v388y2009i9p1849-1856.html

Statistical mechanics approach to a reinforcement learning model with memory

Author

Listed:
  • Lipowski, Adam
  • Gontarek, Krzysztof
  • Ausloos, Marcel

Abstract

We introduce a two-player model of reinforcement learning with memory. Past actions of an iterated game are stored in a memory and used to determine player’s next action. To examine the behaviour of the model some approximate methods are used and confronted against numerical simulations and exact master equation. When the length of memory of players increases to infinity the model undergoes an absorbing-state phase transition. Performance of examined strategies is checked in the prisoner’ dilemma game. It turns out that it is advantageous to have a large memory in symmetric games, but it is better to have a short memory in asymmetric ones.

Suggested Citation

  • Lipowski, Adam & Gontarek, Krzysztof & Ausloos, Marcel, 2009. "Statistical mechanics approach to a reinforcement learning model with memory," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 388(9), pages 1849-1856.
    • Handle: RePEc:eee:phsmap:v:388:y:2009:i:9:p:1849-1856
    DOI: 10.1016/j.physa.2009.01.028
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437109000831
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000
    File URL: https://libkey.io/10.1016/j.physa.2009.01.028?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Drew Fudenberg & Jean Tirole, 1991. "Game Theory," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262061414, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Li, Chanchan & Yang, Zhaoqilin & Jia, Wensheng & Zhao, Hongxin & Wang, Xin, 2026. "Multi-expert policy distillation guided proximal policy optimization for efficient cooperation emergence in spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 202(P1).
    2. Liangliang Chang & Zhipeng Zhang & Chengyi Xia, 2023. "Impact of Decision Feedback on Networked Evolutionary Game with Delays in Control Channel," Dynamic Games and Applications, Springer, vol. 13(3), pages 783-800, September.
    3. Yang, Zhaoqilin & Li, Chanchan & Wang, Xin & Tian, Youliang, 2025. "PPO-ACT: Proximal policy optimization with adversarial curriculum transfer for spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 199(P2).
    4. Yang, Zhaoqilin & Wang, Xin & Zhang, Ruichen & Li, Chanchan & Tian, Youliang, 2025. "TUC-PPO: Team Utility-Constrained Proximal Policy Optimization for spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 199(P3).
    5. Liu, Tianhao, 2021. "A study on day-of-week effect of submission: Based on the data of JSFST," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 563(C).
    6. Kang, Hongwei & Jiang, Chao & Shen, Yong & Sun, Xingping & Chen, Qingyi, 2025. "Neighbor-aware reinforcement learning fosters cooperation in spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 199(P3).
    7. Shen, Yong & Ma, Yujie & Kang, Hongwei & Sun, Xingping & Chen, Qingyi, 2024. "Learning and propagation: Evolutionary dynamics in spatial public goods games through combined Q-learning and Fermi rule," Chaos, Solitons & Fractals, Elsevier, vol. 187(C).
    8. Ausloos, Marcel & Nedic, Olgica & Dekanski, Aleksandar, 2016. "Day of the week effect in paper submission/acceptance/rejection to/in/by peer review journals," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 456(C), pages 197-203.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Janvier D. Nkurunziza, 2005. "Reputation and Credit without Collateral in Africa`s Formal Banking," Economics Series Working Papers WPS/2005-02, University of Oxford, Department of Economics.
    2. Johannes Hörner & Larry Samuelson, 2011. "Managing Strategic Buyers," Journal of Political Economy, University of Chicago Press, vol. 119(3), pages 379-425.
    3. Hugh-Jones, David & Reinstein, David, 2012. "Anonymous rituals," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 478-489.
    4. Kessing, Sebastian G. & Konrad, Kai A. & Kotsogiannis, Christos, 2006. "Federal tax autonomy and the limits of cooperation," Journal of Urban Economics, Elsevier, vol. 59(2), pages 317-329, March.
    5. Schmitz, Patrick W., 2007. "Optimal selling strategies when buyers may have hard information," European Economic Review, Elsevier, vol. 51(4), pages 859-870, May.
    6. Gergely Biczók & Sándor Kardos & Tuan Trinh, 2010. "Pricing Internet access in the presence of user loyalty," Netnomics, Springer, vol. 11(2), pages 119-147, July.
    7. Straub, Paul G., 1995. "Risk dominance and coordination failures in static games," The Quarterly Review of Economics and Finance, Elsevier, vol. 35(4), pages 339-363.
    8. Sandy Fréret & Denis Maguain, 2017. "The effects of agglomeration on tax competition: evidence from a two-regime spatial panel model on French data," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 24(6), pages 1100-1140, December.
    9. , & ,, 2015. "A folk theorem for stochastic games with infrequent state changes," Theoretical Economics, Econometric Society, vol. 10(1), January.
    10. Carlo Rosa & Giovanni Verga, 2006. "The Impact of Central Bank Announcements on Asset Prices in Real Time: Testing the Efficiency of the Euribor Futures Market," CEP Discussion Papers dp0764, Centre for Economic Performance, LSE.
    11. Gintis, Herbert, 2004. "Modeling cooperation among self-interested agents: a critique," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(6), pages 695-714, December.
    12. Clements, Matthew T., 2003. "Precautionary incentives for privately informed victims," International Review of Law and Economics, Elsevier, vol. 23(3), pages 237-251, September.
    13. Jin Cao & Loran Chollete, 2013. "Monetary policy and financial stability in the long run," Working Paper 2013/21, Norges Bank.
    14. Marco Bassetto, 2002. "A Game-Theoretic View of the Fiscal Theory of the Price Level," Econometrica, Econometric Society, vol. 70(6), pages 2167-2195, November.
    15. Casey Rothschild & Florian Scheuer, 2014. "A Theory of Income Taxation under Multidimensional Skill Heterogeneity," NBER Working Papers 19822, National Bureau of Economic Research, Inc.
    16. Arthur Schram & Boris Van Leeuwen & Theo Offerman, 2013. "Superstars Need Social Benefits: An Experiment on Network Formation," Working Papers 1306, Departament Empresa, Universitat Autònoma de Barcelona, revised Jul 2013.
    17. Patrick W. Schmitz, 2006. "Book Review," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 162(3), pages 535-542, September.
    18. Van Long, Ngo & Shimomura, Koji, 1998. "Some results on the Markov equilibria of a class of homogeneous differential games," Journal of Economic Behavior & Organization, Elsevier, vol. 33(3-4), pages 557-566, January.
    19. Strausz, Roland, 2006. "Deterministic versus stochastic mechanisms in principal-agent models," Journal of Economic Theory, Elsevier, vol. 128(1), pages 306-314, May.
    20. Ravi Anupindi & Yehuda Bassok & Eitan Zemel, 2001. "A General Framework for the Study of Decentralized Distribution Systems," Manufacturing & Service Operations Management, INFORMS, vol. 3(4), pages 349-368, February.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:388:y:2009:i:9:p:1849-1856. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.