IDEAS home Printed from https://ideas.repec.org/a/inm/ormoor/v41y2016i4p1297-1324.html
   My bibliography  Save this article

Learning in Games via Reinforcement and Regularization

Author

Listed:
  • Panayotis Mertikopoulos

    (CNRS (French National Center for Scientific Research), LIG, F-38000 Grenoble, France; and University Grenoble Alpes, LIG, F-38000 Grenoble, France)

  • William H. Sandholm

    (Department of Economics, University of Wisconsin, Madison Wisconsin 53706)

Abstract

We investigate a class of reinforcement learning dynamics where players adjust their strategies based on their actions’ cumulative payoffs over time—specifically, by playing mixed strategies that maximize their expected cumulative payoff minus a regularization term. A widely studied example is exponential reinforcement learning, a process induced by an entropic regularization term which leads mixed strategies to evolve according to the replicator dynamics. However, in contrast to the class of regularization functions used to define smooth best responses in models of stochastic fictitious play, the functions used in this paper need not be infinitely steep at the boundary of the simplex; in fact, dropping this requirement gives rise to an important dichotomy between steep and nonsteep cases. In this general framework, we extend several properties of exponential learning, including the elimination of dominated strategies, the asymptotic stability of strict Nash equilibria, and the convergence of time-averaged trajectories in zero-sum games with an interior Nash equilibrium.

Suggested Citation

  • Panayotis Mertikopoulos & William H. Sandholm, 2016. "Learning in Games via Reinforcement and Regularization," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1297-1324, November.
  • Handle: RePEc:inm:ormoor:v:41:y:2016:i:4:p:1297-1324
    DOI: 10.1287/moor.2016.0778
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/moor.2016.0778
    Download Restriction: no

    File URL: https://libkey.io/10.1287/moor.2016.0778?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Michel Benaïm & Josef Hofbauer & Sylvain Sorin, 2006. "Stochastic Approximations and Differential Inclusions, Part II: Applications," Mathematics of Operations Research, INFORMS, vol. 31(4), pages 673-695, November.
    2. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    3. repec:dau:papers:123456789/1014 is not listed on IDEAS
    4. Cominetti, Roberto & Melo, Emerson & Sorin, Sylvain, 2010. "A payoff-based learning procedure and its application to traffic games," Games and Economic Behavior, Elsevier, vol. 70(1), pages 71-83, September.
    5. Viossat, Yannick & Zapechelnyuk, Andriy, 2013. "No-regret dynamics and fictitious play," Journal of Economic Theory, Elsevier, vol. 148(2), pages 825-842.
    6. Hopkins, Ed & Posch, Martin, 2005. "Attainability of boundary points under reinforcement learning," Games and Economic Behavior, Elsevier, vol. 53(1), pages 110-125, October.
    7. Hopkins, Ed, 1999. "Learning, Matching, and Aggregation," Games and Economic Behavior, Elsevier, vol. 26(1), pages 79-110, January.
    8. Sandholm, William H. & DokumacI, Emin & Lahkar, Ratul, 2008. "The projection dynamic and the replicator dynamic," Games and Economic Behavior, Elsevier, vol. 64(2), pages 666-683, November.
    9. Martin Posch, 1997. "Cycling in a stochastic learning algorithm for normal form games," Journal of Evolutionary Economics, Springer, vol. 7(2), pages 193-207.
    10. McKelvey Richard D. & Palfrey Thomas R., 1995. "Quantal Response Equilibria for Normal Form Games," Games and Economic Behavior, Elsevier, vol. 10(1), pages 6-38, July.
    11. Ed Hopkins, 2002. "Two Competing Models of How People Learn in Games," Econometrica, Econometric Society, vol. 70(6), pages 2141-2166, November.
    12. Josef Hofbauer & William H. Sandholm, 2002. "On the Global Convergence of Stochastic Fictitious Play," Econometrica, Econometric Society, vol. 70(6), pages 2265-2294, November.
    13. Sergiu Hart & Andreu Mas-Colell, 2013. "A Simple Adaptive Procedure Leading To Correlated Equilibrium," World Scientific Book Chapters, in: Simple Adaptive Strategies From Regret-Matching to Uncoupled Dynamics, chapter 2, pages 17-46, World Scientific Publishing Co. Pte. Ltd..
    14. Borgers, Tilman & Sarin, Rajiv, 1997. "Learning Through Reinforcement and Replicator Dynamics," Journal of Economic Theory, Elsevier, vol. 77(1), pages 1-14, November.
    15. Anna Nagurney & Ding Zhang, 1997. "Projected Dynamical Systems in the Formulation, Stability Analysis, and Computation of Fixed-Demand Traffic Network Equilibria," Transportation Science, INFORMS, vol. 31(2), pages 147-158, May.
    16. Samuelson, Larry & Zhang, Jianbo, 1992. "Evolutionary stability in asymmetric games," Journal of Economic Theory, Elsevier, vol. 57(2), pages 363-391, August.
    17. Drew Fudenberg & David K. Levine, 1998. "The Theory of Learning in Games," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262061945.
    18. Hopkins, Ed, 1999. "A Note on Best Response Dynamics," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 138-150, October.
    19. Rustichini, Aldo, 1999. "Optimal Properties of Stimulus--Response Learning Models," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 244-273, October.
    20. Beggs, A.W., 2005. "On the convergence of reinforcement learning," Journal of Economic Theory, Elsevier, vol. 122(1), pages 1-36, May.
    21. Michel Benaim & Josef Hofbauer & Sylvain Sorin, 2005. "Stochastic Approximations and Differential Inclusions II: Applications," Levine's Bibliography 784828000000000098, UCLA Department of Economics.
    22. Pierre Coucheney & Bruno Gaujal & Panayotis Mertikopoulos, 2015. "Penalty-Regulated Dynamics and Robust Learning Procedures in Games," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 611-633, March.
    23. Friedman, Daniel, 1991. "Evolutionary Games in Economics," Econometrica, Econometric Society, vol. 59(3), pages 637-666, May.
    24. Josef Hofbauer & Sylvain Sorin & Yannick Viossat, 2009. "Time Average Replicator and Best-Reply Dynamics," Mathematics of Operations Research, INFORMS, vol. 34(2), pages 263-269, May.
    25. Laraki, Rida & Mertikopoulos, Panayotis, 2013. "Higher order game dynamics," Journal of Economic Theory, Elsevier, vol. 148(6), pages 2666-2695.
    26. Lahkar, Ratul & Sandholm, William H., 2008. "The projection dynamic and the geometry of population games," Games and Economic Behavior, Elsevier, vol. 64(2), pages 565-590, November.
    27. John G. Cross, 1973. "A Stochastic Learning Model of Economic Behavior," The Quarterly Journal of Economics, Oxford University Press, vol. 87(2), pages 239-266.
    28. Freund, Yoav & Schapire, Robert E., 1999. "Adaptive Game Playing Using Multiplicative Weights," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 79-103, October.
    29. Nachbar, J H, 1990. ""Evolutionary" Selection Dynamics in Games: Convergence and Limit Properties," International Journal of Game Theory, Springer;Game Theory Society, vol. 19(1), pages 59-89.
    30. Krzysztof C. Kiwiel, 1997. "Free-Steering Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints," Mathematics of Operations Research, INFORMS, vol. 22(2), pages 326-349, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bravo, Mario & Mertikopoulos, Panayotis, 2017. "On the robustness of learning in games with stochastically perturbed payoff observations," Games and Economic Behavior, Elsevier, vol. 103(C), pages 41-66.
    2. Mertikopoulos, Panayotis & Sandholm, William H., 2018. "Riemannian game dynamics," Journal of Economic Theory, Elsevier, vol. 177(C), pages 315-364.
    3. Jonathan Newton, 2018. "Evolutionary Game Theory: A Renaissance," Games, MDPI, Open Access Journal, vol. 9(2), pages 1-67, May.
    4. Kuang Xu & Se-Young Yun, 2020. "Reinforcement with Fading Memories," Mathematics of Operations Research, INFORMS, vol. 45(4), pages 1258-1288, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sandholm, William H., 2015. "Population Games and Deterministic Evolutionary Dynamics," Handbook of Game Theory with Economic Applications,, Elsevier.
    2. Jonathan Newton, 2018. "Evolutionary Game Theory: A Renaissance," Games, MDPI, Open Access Journal, vol. 9(2), pages 1-67, May.
    3. Mertikopoulos, Panayotis & Sandholm, William H., 2018. "Riemannian game dynamics," Journal of Economic Theory, Elsevier, vol. 177(C), pages 315-364.
    4. Bravo, Mario & Mertikopoulos, Panayotis, 2017. "On the robustness of learning in games with stochastically perturbed payoff observations," Games and Economic Behavior, Elsevier, vol. 103(C), pages 41-66.
    5. Benaïm, Michel & Hofbauer, Josef & Hopkins, Ed, 2009. "Learning in games with unstable equilibria," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1694-1709, July.
    6. Pierre Coucheney & Bruno Gaujal & Panayotis Mertikopoulos, 2015. "Penalty-Regulated Dynamics and Robust Learning Procedures in Games," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 611-633, March.
    7. Cominetti, Roberto & Melo, Emerson & Sorin, Sylvain, 2010. "A payoff-based learning procedure and its application to traffic games," Games and Economic Behavior, Elsevier, vol. 70(1), pages 71-83, September.
    8. Cason, Timothy N. & Friedman, Daniel & Hopkins, Ed, 2010. "Testing the TASP: An experimental investigation of learning in games with unstable equilibria," Journal of Economic Theory, Elsevier, vol. 145(6), pages 2309-2331, November.
    9. Naoki Funai, 2019. "Convergence results on stochastic adaptive learning," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 68(4), pages 907-934, November.
    10. Mario Bravo & Mathieu Faure, 2013. "Reinforcement Learning with Restrictions on the Action Set," AMSE Working Papers 1335, Aix-Marseille School of Economics, France, revised 01 Jul 2013.
    11. Izquierdo, Luis R. & Izquierdo, Segismundo S. & Gotts, Nicholas M. & Polhill, J. Gary, 2007. "Transient and asymptotic dynamics of reinforcement learning in games," Games and Economic Behavior, Elsevier, vol. 61(2), pages 259-276, November.
    12. Ianni, Antonella, 2014. "Learning strict Nash equilibria through reinforcement," Journal of Mathematical Economics, Elsevier, vol. 50(C), pages 148-155.
    13. Hofbauer, Josef & Hopkins, Ed, 2005. "Learning in perturbed asymmetric games," Games and Economic Behavior, Elsevier, vol. 52(1), pages 133-152, July.
    14. Oyarzun, Carlos & Sarin, Rajiv, 2013. "Learning and risk aversion," Journal of Economic Theory, Elsevier, vol. 148(1), pages 196-225.
    15. Leslie, David S. & Collins, E.J., 2006. "Generalised weakened fictitious play," Games and Economic Behavior, Elsevier, vol. 56(2), pages 285-298, August.
    16. Bernergård, Axel & Mohlin, Erik, 2019. "Evolutionary selection against iteratively weakly dominated strategies," Games and Economic Behavior, Elsevier, vol. 117(C), pages 82-97.
    17. Andriy Zapechelnyuk, 2009. "Limit Behavior of No-regret Dynamics," Discussion Papers 21, Kyiv School of Economics.
    18. Reinoud Joosten & Berend Roorda, 2008. "Generalized projection dynamics in evolutionary game theory," Papers on Economics and Evolution 2008-11, Philipps University Marburg, Department of Geography.
    19. Yannick Viossat, 2015. "Evolutionary dynamics and dominated strategies," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 3(1), pages 91-113, April.
    20. Josephson, Jens, 2008. "A numerical analysis of the evolutionary stability of learning rules," Journal of Economic Dynamics and Control, Elsevier, vol. 32(5), pages 1569-1599, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormoor:v:41:y:2016:i:4:p:1297-1324. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/inforea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Matthew Walls (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.