IDEAS home Printed from https://ideas.repec.org/a/inm/ormoor/v41y2016i4p1297-1324.html
   My bibliography  Save this article

Learning in Games via Reinforcement and Regularization

Author

Listed:
  • Panayotis Mertikopoulos

    () (CNRS (French National Center for Scientific Research), LIG, F-38000 Grenoble, France; and University Grenoble Alpes, LIG, F-38000 Grenoble, France)

  • William H. Sandholm

    () (Department of Economics, University of Wisconsin, Madison Wisconsin 53706)

Abstract

We investigate a class of reinforcement learning dynamics where players adjust their strategies based on their actions’ cumulative payoffs over time—specifically, by playing mixed strategies that maximize their expected cumulative payoff minus a regularization term. A widely studied example is exponential reinforcement learning, a process induced by an entropic regularization term which leads mixed strategies to evolve according to the replicator dynamics. However, in contrast to the class of regularization functions used to define smooth best responses in models of stochastic fictitious play, the functions used in this paper need not be infinitely steep at the boundary of the simplex; in fact, dropping this requirement gives rise to an important dichotomy between steep and nonsteep cases. In this general framework, we extend several properties of exponential learning, including the elimination of dominated strategies, the asymptotic stability of strict Nash equilibria, and the convergence of time-averaged trajectories in zero-sum games with an interior Nash equilibrium.

Suggested Citation

  • Panayotis Mertikopoulos & William H. Sandholm, 2016. "Learning in Games via Reinforcement and Regularization," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1297-1324, November.
  • Handle: RePEc:inm:ormoor:v:41:y:2016:i:4:p:1297-1324
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/moor.2016.0778
    Download Restriction: no

    References listed on IDEAS

    as
    1. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    2. repec:dau:papers:123456789/1014 is not listed on IDEAS
    3. Cominetti, Roberto & Melo, Emerson & Sorin, Sylvain, 2010. "A payoff-based learning procedure and its application to traffic games," Games and Economic Behavior, Elsevier, vol. 70(1), pages 71-83, September.
    4. Viossat, Yannick & Zapechelnyuk, Andriy, 2013. "No-regret dynamics and fictitious play," Journal of Economic Theory, Elsevier, vol. 148(2), pages 825-842.
    5. Hopkins, Ed & Posch, Martin, 2005. "Attainability of boundary points under reinforcement learning," Games and Economic Behavior, Elsevier, vol. 53(1), pages 110-125, October.
    6. Hopkins, Ed, 1999. "Learning, Matching, and Aggregation," Games and Economic Behavior, Elsevier, vol. 26(1), pages 79-110, January.
    7. Sandholm, William H. & DokumacI, Emin & Lahkar, Ratul, 2008. "The projection dynamic and the replicator dynamic," Games and Economic Behavior, Elsevier, vol. 64(2), pages 666-683, November.
    8. Martin Posch, 1997. "Cycling in a stochastic learning algorithm for normal form games," Journal of Evolutionary Economics, Springer, vol. 7(2), pages 193-207.
    9. McKelvey Richard D. & Palfrey Thomas R., 1995. "Quantal Response Equilibria for Normal Form Games," Games and Economic Behavior, Elsevier, vol. 10(1), pages 6-38, July.
    10. Ed Hopkins, 2002. "Two Competing Models of How People Learn in Games," Econometrica, Econometric Society, vol. 70(6), pages 2141-2166, November.
    11. Josef Hofbauer & William H. Sandholm, 2002. "On the Global Convergence of Stochastic Fictitious Play," Econometrica, Econometric Society, vol. 70(6), pages 2265-2294, November.
    12. Borgers, Tilman & Sarin, Rajiv, 1997. "Learning Through Reinforcement and Replicator Dynamics," Journal of Economic Theory, Elsevier, vol. 77(1), pages 1-14, November.
    13. Anna Nagurney & Ding Zhang, 1997. "Projected Dynamical Systems in the Formulation, Stability Analysis, and Computation of Fixed-Demand Traffic Network Equilibria," Transportation Science, INFORMS, vol. 31(2), pages 147-158, May.
    14. Samuelson, Larry & Zhang, Jianbo, 1992. "Evolutionary stability in asymmetric games," Journal of Economic Theory, Elsevier, vol. 57(2), pages 363-391, August.
    15. Drew Fudenberg & David K. Levine, 1998. "The Theory of Learning in Games," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262061945, October.
    16. Hopkins, Ed, 1999. "A Note on Best Response Dynamics," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 138-150, October.
    17. Rustichini, Aldo, 1999. "Optimal Properties of Stimulus--Response Learning Models," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 244-273, October.
    18. Beggs, A.W., 2005. "On the convergence of reinforcement learning," Journal of Economic Theory, Elsevier, vol. 122(1), pages 1-36, May.
    19. Michel Benaim & Josef Hofbauer & Sylvain Sorin, 2005. "Stochastic Approximations and Differential Inclusions II: Applications," Levine's Bibliography 784828000000000098, UCLA Department of Economics.
    20. Pierre Coucheney & Bruno Gaujal & Panayotis Mertikopoulos, 2015. "Penalty-Regulated Dynamics and Robust Learning Procedures in Games," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 611-633, March.
    21. Friedman, Daniel, 1991. "Evolutionary Games in Economics," Econometrica, Econometric Society, vol. 59(3), pages 637-666, May.
    22. Michel Benaïm & Josef Hofbauer & Sylvain Sorin, 2005. "Stochastic Approximations and Differential Inclusions; Part II: Applications," Working Papers hal-00242974, HAL.
    23. Josef Hofbauer & Sylvain Sorin & Yannick Viossat, 2009. "Time Average Replicator and Best-Reply Dynamics," Mathematics of Operations Research, INFORMS, vol. 34(2), pages 263-269, May.
    24. Laraki, Rida & Mertikopoulos, Panayotis, 2013. "Higher order game dynamics," Journal of Economic Theory, Elsevier, vol. 148(6), pages 2666-2695.
    25. Lahkar, Ratul & Sandholm, William H., 2008. "The projection dynamic and the geometry of population games," Games and Economic Behavior, Elsevier, vol. 64(2), pages 565-590, November.
    26. John G. Cross, 1973. "A Stochastic Learning Model of Economic Behavior," The Quarterly Journal of Economics, Oxford University Press, vol. 87(2), pages 239-266.
    27. Freund, Yoav & Schapire, Robert E., 1999. "Adaptive Game Playing Using Multiplicative Weights," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 79-103, October.
    28. Nachbar, J H, 1990. ""Evolutionary" Selection Dynamics in Games: Convergence and Limit Properties," International Journal of Game Theory, Springer;Game Theory Society, vol. 19(1), pages 59-89.
    29. Krzysztof C. Kiwiel, 1997. "Free-Steering Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints," Mathematics of Operations Research, INFORMS, vol. 22(2), pages 326-349, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:eee:gamebe:v:103:y:2017:i:c:p:41-66 is not listed on IDEAS
    2. repec:eee:jetheo:v:177:y:2018:i:c:p:315-364 is not listed on IDEAS

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormoor:v:41:y:2016:i:4:p:1297-1324. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mirko Janc) The email address of this maintainer does not seem to be valid anymore. Please ask Mirko Janc to update the entry or send us the correct email address. General contact details of provider: http://edirc.repec.org/data/inforea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.