IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1502.06901.html
   My bibliography  Save this paper

Equilibrium in Misspecified Markov Decision Processes

Author

Listed:
  • Ignacio Esponda
  • Demian Pouzo

Abstract

We study Markov decision problems where the agent does not know the transition probability function mapping current states and actions to future states. The agent has a prior belief over a set of possible transition functions and updates beliefs using Bayes' rule. We allow her to be misspecified in the sense that the true transition probability function is not in the support of her prior. This problem is relevant in many economic settings but is usually not amenable to analysis by the researcher. We make the problem tractable by studying asymptotic behavior. We propose an equilibrium notion and provide conditions under which it characterizes steady state behavior. In the special case where the problem is static, equilibrium coincides with the single-agent version of Berk-Nash equilibrium (Esponda and Pouzo (2016)). We also discuss subtle issues that arise exclusively in dynamic settings due to the possibility of a negative value of experimentation.

Suggested Citation

  • Ignacio Esponda & Demian Pouzo, 2015. "Equilibrium in Misspecified Markov Decision Processes," Papers 1502.06901, arXiv.org, revised May 2016.
  • Handle: RePEc:arx:papers:1502.06901
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1502.06901
    File Function: Latest version
    Download Restriction: no

    References listed on IDEAS

    as
    1. Doraszelski, Ulrich & Escobar, Juan, 2010. "A theory of regular Markov perfect equilibria in dynamic stochastic games: genericity, stability, and purification," Theoretical Economics, Econometric Society, vol. 5(3), September.
    2. Erik Eyster & Matthew Rabin, 2005. "Cursed Equilibrium," Econometrica, Econometric Society, vol. 73(5), pages 1623-1672, September.
    3. Dekel, Eddie & Fudenberg, Drew & Levine, David K., 2004. "Learning to play Bayesian games," Games and Economic Behavior, Elsevier, vol. 46(2), pages 282-303, February.
    4. Enriqueta Aragones & Itzhak Gilboa & Andrew Postlewaite & David Schmeidler, 2005. "Fact-Free Learning," American Economic Review, American Economic Association, vol. 95(5), pages 1355-1368, December.
    5. Philippe Aghion & Patrick Bolton & Christopher Harris & Bruno Jullien, 1991. "Optimal Learning by Experimentation," Review of Economic Studies, Oxford University Press, vol. 58(4), pages 621-654.
    6. Jehiel, Philippe, 2005. "Analogy-based expectation equilibrium," Journal of Economic Theory, Elsevier, vol. 123(2), pages 81-104, August.
    7. Fudenberg, Drew & Levine, David K, 1993. "Steady State Learning and Nash Equilibrium," Econometrica, Econometric Society, vol. 61(3), pages 547-573, May.
    8. Blume, Lawrence E. & Easley, David, 1984. "Rational expectations equilibrium: An alternative approach," Journal of Economic Theory, Elsevier, vol. 34(1), pages 116-129, October.
    9. Kalai, Ehud & Lehrer, Ehud, 1993. "Rational Learning Leads to Nash Equilibrium," Econometrica, Econometric Society, vol. 61(5), pages 1019-1045, September.
    10. Jehiel, Philippe & Samet, Dov, 2007. "Valuation equilibrium," Theoretical Economics, Econometric Society, vol. 2(2), June.
    11. repec:hrv:faseco:30747159 is not listed on IDEAS
    12. Osborne, Martin J & Rubinstein, Ariel, 1998. "Games with Procedurally Rational Players," American Economic Review, American Economic Association, vol. 88(4), pages 834-847, September.
    13. Nabil I. Al-Najjar, 2009. "Decision Makers as Statisticians: Diversity, Ambiguity, and Learning," Econometrica, Econometric Society, vol. 77(5), pages 1371-1401, September.
    14. Fudenberg, Drew & Levine, David K, 1993. "Self-Confirming Equilibrium," Econometrica, Econometric Society, vol. 61(3), pages 523-545, May.
    15. Michele Piccione & Ariel Rubinstein, 2003. "Modeling the Economic Interaction of Agents With Diverse Abilities to Recognize Equilibrium Patterns," Journal of the European Economic Association, MIT Press, vol. 1(1), pages 212-223, March.
    16. Nyarko, Yaw, 1991. "Learning in mis-specified models and the possibility of cycles," Journal of Economic Theory, Elsevier, vol. 55(2), pages 416-427, December.
    17. Sobel, Joel, 1984. "Non-linear prices and price-taking behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 5(3-4), pages 387-396.
    18. Ignacio Esponda, 2008. "Behavioral Equilibrium in Economies with Adverse Selection," American Economic Review, American Economic Association, vol. 98(4), pages 1269-1291, September.
    19. Barberis, Nicholas & Shleifer, Andrei & Vishny, Robert, 1998. "A model of investor sentiment," Journal of Financial Economics, Elsevier, vol. 49(3), pages 307-343, September.
    20. Rothschild, Michael, 1974. "A two-armed bandit theory of market pricing," Journal of Economic Theory, Elsevier, vol. 9(2), pages 185-202, October.
    21. McLennan, Andrew, 1984. "Price dispersion and incomplete learning in the long run," Journal of Economic Dynamics and Control, Elsevier, vol. 7(3), pages 331-347, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fudenberg, Drew & Romanyuk, Gleb & Strack, Philipp, 2017. "Active learning with a misspecified prior," Theoretical Economics, Econometric Society, vol. 12(3), September.
    2. Ignacio Esponda & Demian Pouzo & Yuichi Yamamoto, 2019. "Asymptotic Behavior of Bayesian Learners with Misspecified Models," Papers 1904.08551, arXiv.org, revised Oct 2019.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1502.06901. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (arXiv administrators). General contact details of provider: http://arxiv.org/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.