IDEAS home Printed from https://ideas.repec.org/a/spr/dyngam/v13y2023i1d10.1007_s13235-022-00450-2.html
   My bibliography  Save this article

Q-Learning in Regularized Mean-field Games

Author

Listed:
  • Berkay Anahtarci

    (Özyeğin University)

  • Can Deha Kariksiz

    (Özyeğin University)

  • Naci Saldi

    (Bilkent University)

Abstract

In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. Regularization is introduced by adding a strongly concave regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. The regularization term in general makes reinforcement learning algorithm more robust to the system components. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

Suggested Citation

  • Berkay Anahtarci & Can Deha Kariksiz & Naci Saldi, 2023. "Q-Learning in Regularized Mean-field Games," Dynamic Games and Applications, Springer, vol. 13(1), pages 89-117, March.
    • Handle: RePEc:spr:dyngam:v:13:y:2023:i:1:d:10.1007_s13235-022-00450-2
    DOI: 10.1007/s13235-022-00450-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s13235-022-00450-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s13235-022-00450-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. A. Bensoussan & K. Sung & S. Yam, 2013. "Linear–Quadratic Time-Inconsistent Mean Field Games," Dynamic Games and Applications, Springer, vol. 3(4), pages 537-552, December.
    2. Naci Saldi & Tamer Başar & Maxim Raginsky, 2019. "Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions," Mathematics of Operations Research, INFORMS, vol. 44(3), pages 1006-1033, August.
    3. Adlakha, Sachin & Johari, Ramesh & Weintraub, Gabriel Y., 2015. "Equilibria of dynamic games with many players: Existence, approximation, and market structure," Journal of Economic Theory, Elsevier, vol. 156(C), pages 269-316.
    4. Piotr Więcek & Eitan Altman, 2015. "Stationary Anonymous Sequential Games with Undiscounted Rewards," Journal of Optimization Theory and Applications, Springer, vol. 166(2), pages 686-710, August.
    5. Diogo Gomes & João Saúde, 2014. "Mean Field Games Models—A Brief Survey," Dynamic Games and Applications, Springer, vol. 4(2), pages 110-154, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Naci Saldi & Tamer Bas¸ ar & Maxim Raginsky, 2020. "Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games," Mathematics of Operations Research, INFORMS, vol. 45(4), pages 1596-1620, November.
    2. Naci Saldi & Tamer Başar & Maxim Raginsky, 2023. "Partially Observed Discrete-Time Risk-Sensitive Mean Field Games," Dynamic Games and Applications, Springer, vol. 13(3), pages 929-960, September.
    3. Piotr Więcek, 2020. "Discrete-Time Ergodic Mean-Field Games with Average Reward on Compact Spaces," Dynamic Games and Applications, Springer, vol. 10(1), pages 222-256, March.
    4. Piotr Więcek, 2017. "Total Reward Semi-Markov Mean-Field Games with Complementarity Properties," Dynamic Games and Applications, Springer, vol. 7(3), pages 507-529, September.
    5. Naci Saldi & Tamer Başar & Maxim Raginsky, 2019. "Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions," Mathematics of Operations Research, INFORMS, vol. 44(3), pages 1006-1033, August.
    6. Régis Chenavaz & Corina Paraschiv & Gabriel Turinici, 2021. "Dynamic Pricing of New Products in Competitive Markets: A Mean-Field Game Approach," Dynamic Games and Applications, Springer, vol. 11(3), pages 463-490, September.
    7. Paulo B. Brito, 2022. "The dynamics of growth and distribution in a spatially heterogeneous world," Portuguese Economic Journal, Springer;Instituto Superior de Economia e Gestao, vol. 21(3), pages 311-350, September.
    8. Régis Chenavaz & Corina Paraschiv & Gabriel Turinici, 2017. "Dynamic Pricing of New Products in Competitive Markets: A Mean-Field Game Approach," Working Papers hal-01592958, HAL.
    9. Vassili N. Kolokoltsov, 2021. "Quantum Mean-Field Games with the Observations of Counting Type," Games, MDPI, vol. 12(1), pages 1-14, January.
    10. Qinglong Zhou & Gaofeng Zong, 2016. "Time-Inconsistent Stochastic Linear-quadratic Differential Game," Papers 1607.00638, arXiv.org.
    11. Alain Bensoussan & Guiyuan Ma & Chi Chung Siu & Sheung Chi Phillip Yam, 2022. "Dynamic mean–variance problem with frictions," Finance and Stochastics, Springer, vol. 26(2), pages 267-300, April.
    12. Ezzat Elokda & Andrea Censi & Saverio Bolognani, 2021. "Dynamic population games," Papers 2104.14662, arXiv.org.
    13. Hoang, Lê Nguyên & Soumis, François & Zaccour, Georges, 2019. "The return function: A new computable perspective on Bayesian–Nash equilibria," European Journal of Operational Research, Elsevier, vol. 279(2), pages 471-485.
    14. Haoyang Cao & Jodi Dianetti & Giorgio Ferrari, 2021. "Stationary Discounted and Ergodic Mean Field Games of Singular Control," Papers 2105.07213, arXiv.org.
    15. Diogo A. Gomes & João Saúde, 2021. "A Mean-Field Game Approach to Price Formation," Dynamic Games and Applications, Springer, vol. 11(1), pages 29-53, March.
    16. Balbus, Lukasz & Dziewulski, Pawel & Reffett, Kevin & Wozny, Lukasz, 2022. "Markov distributional equilibrium dynamics in games with complementarities and no aggregate risk," Theoretical Economics, Econometric Society, vol. 17(2), May.
    17. Zongxia Liang & Keyu Zhang, 2023. "Time-inconsistent mean field and n-agent games under relative performance criteria," Papers 2312.14437, arXiv.org.
    18. Shuzhen Yang, 2020. "Bellman type strategy for the continuous time mean-variance model," Papers 2005.01904, arXiv.org, revised Jul 2020.
    19. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    20. Deepanshu Vasal, 2023. "Social herding in mean field games," Papers 2303.03303, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:dyngam:v:13:y:2023:i:1:d:10.1007_s13235-022-00450-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.