IDEAS home Printed from https://ideas.repec.org/a/spr/finsto/v25y2021i2d10.1007_s00780-020-00443-2.html
   My bibliography  Save this article

Markov decision processes with quasi-hyperbolic discounting

Author

Listed:
  • Anna Jaśkiewicz

    (Wrocław University of Science and Technology)

  • Andrzej S. Nowak

    (Faculty of Mathematics, Computer Science and Econometrics, University of Zielona Góra)

Abstract

We study Markov decision processes with Borel state spaces under quasi-hyperbolic discounting. This type of discounting nicely models human behaviour, which is time-inconsistent in the long run. The decision maker has preferences changing in time. Therefore, the standard approach based on the Bellman optimality principle fails. Within a dynamic game-theoretic framework, we prove the existence of randomised stationary Markov perfect equilibria for a large class of Markov decision processes with transitions having a density function. We also show that randomisation can be restricted to two actions in every state of the process. Moreover, we prove that under some conditions, this equilibrium can be replaced by a deterministic one. For models with countable state spaces, we establish the existence of deterministic Markov perfect equilibria. Many examples are given to illustrate our results, including a portfolio selection model with quasi-hyperbolic discounting.

Suggested Citation

  • Anna Jaśkiewicz & Andrzej S. Nowak, 2021. "Markov decision processes with quasi-hyperbolic discounting," Finance and Stochastics, Springer, vol. 25(2), pages 189-229, April.
    • Handle: RePEc:spr:finsto:v:25:y:2021:i:2:d:10.1007_s00780-020-00443-2
    DOI: 10.1007/s00780-020-00443-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00780-020-00443-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00780-020-00443-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Caballero, Ricardo J., 1990. "Consumption puzzles and precautionary savings," Journal of Monetary Economics, Elsevier, vol. 25(1), pages 113-136, January.
    2. Per Krusell & Anthony A. Smith, Jr., 2003. "Consumption--Savings Decisions with Quasi--Geometric Discounting," Econometrica, Econometric Society, vol. 71(1), pages 365-375, January.
    3. John Stachurski, 2009. "Economic Dynamics: Theory and Computation," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262012774, December.
    4. Krusell, Per & Kuruscu, Burhanettin & Smith, Anthony Jr., 2002. "Equilibrium Welfare and Government Policy with Quasi-geometric Discounting," Journal of Economic Theory, Elsevier, vol. 105(1), pages 42-72, July.
    5. Chatterjee, Satyajit & Eyigungor, Burcu, 2016. "Continuous Markov equilibria with quasi-geometric discounting," Journal of Economic Theory, Elsevier, vol. 163(C), pages 467-494.
    6. Łukasz Balbus & Anna Jaśkiewicz & Andrzej S. Nowak, 2020. "Markov perfect equilibria in a dynamic decision model with quasi-hyperbolic discounting," Annals of Operations Research, Springer, vol. 287(2), pages 573-591, April.
    7. John Duggan, 2012. "Noisy Stochastic Games," Econometrica, Econometric Society, vol. 80(5), pages 2017-2045, September.
    8. Tomas Björk & Agatha Murgoci, 2014. "A theory of Markovian time-inconsistent stochastic control in discrete time," Finance and Stochastics, Springer, vol. 18(3), pages 545-592, July.
    9. Bezalel Peleg & Menahem E. Yaari, 1973. "On the Existence of a Consistent Course of Action when Tastes are Changing," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 40(3), pages 391-401.
    10. Erzo G. J. Luttmer & Thomas Mariotti, 2003. "Subjective Discounting in an Exchange Economy," Journal of Political Economy, University of Chicago Press, vol. 111(5), pages 959-989, October.
    11. He, Wei & Sun, Yeneng, 2017. "Stationary Markov perfect equilibria in discounted stochastic games," Journal of Economic Theory, Elsevier, vol. 169(C), pages 35-61.
    12. Harris, Christopher & Laibson, David, 2001. "Dynamic Choices of Hyperbolic Consumers," Econometrica, Econometric Society, vol. 69(4), pages 935-957, July.
    13. Jaśkiewicz, Anna & Nowak, Andrzej S., 2014. "Stationary Markov perfect equilibria in risk sensitive stochastic overlapping generations models," Journal of Economic Theory, Elsevier, vol. 151(C), pages 411-447.
    14. Merton, Robert C., 1971. "Optimum consumption and portfolio rules in a continuous-time model," Journal of Economic Theory, Elsevier, vol. 3(4), pages 373-413, December.
    15. John Duggan, 2012. "Noisy Stochastic Games," RCER Working Papers 570, University of Rochester - Center for Economic Research (RCER).
    16. E. S. Phelps & R. A. Pollak, 1968. "On Second-Best National Saving and Game-Equilibrium Growth," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 35(2), pages 185-199.
    17. Lilia Maliar & Serguei Maliar, 2004. "Quasi‐geometric discounting: A closed‐form solution under the exponential utility function," Bulletin of Economic Research, Wiley Blackwell, vol. 56(2), pages 201-206, April.
    18. Nicolas Vieille & Jörgen Weibull, 2009. "Multiple solutions under quasi-exponential discounting," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 39(3), pages 513-526, June.
    19. Wei He & Yeneng Sun, 2018. "Conditional expectation of correspondences and economic applications," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 66(2), pages 265-299, August.
    20. Balbus, Łukasz & Reffett, Kevin & Woźny, Łukasz, 2018. "On uniqueness of time-consistent Markov policies for quasi-hyperbolic consumers under uncertainty," Journal of Economic Theory, Elsevier, vol. 176(C), pages 293-310.
    21. José Luis Montiel Olea & Tomasz Strzalecki, 2014. "Axiomatization and Measurement of Quasi-Hyperbolic Discounting," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 129(3), pages 1449-1499.
    22. Alain Haurie, 2005. "A Multigenerational Game Model to Analyze Sustainable Development," Annals of Operations Research, Springer, vol. 137(1), pages 369-386, July.
    23. Paul A. Samuelson, 2011. "Lifetime Portfolio Selection by Dynamic Stochastic Programming," World Scientific Book Chapters, in: Leonard C MacLean & Edward O Thorp & William T Ziemba (ed.), THE KELLY CAPITAL GROWTH INVESTMENT CRITERION THEORY and PRACTICE, chapter 31, pages 465-472, World Scientific Publishing Co. Pte. Ltd..
    24. Lilia Maliar & Serguei Maliar, 2016. "Ruling Out Multiplicity of Smooth Equilibria in Dynamic Games: A Hyperbolic Discounting Example," Dynamic Games and Applications, Springer, vol. 6(2), pages 243-261, June.
    25. Łukasz Balbus & Anna Jaśkiewicz & Andrzej S. Nowak, 2014. "Robust Markov Perfect Equilibria in a Dynamic Choice Model with Quasi-hyperbolic Discounting," Dynamic Modeling and Econometrics in Economics and Finance, in: Josef Haunschmied & Vladimir M. Veliov & Stefan Wrzaczek (ed.), Dynamic Games in Economics, edition 127, pages 1-22, Springer.
    26. Bernheim, B. Douglas & Ray, Debraj, 1989. "Markov perfect equilibria in altruistic growth economies with production uncertainty," Journal of Economic Theory, Elsevier, vol. 47(1), pages 195-202, February.
    27. Robert J. Barro, 1999. "Ramsey Meets Laibson in the Neoclassical Growth Model," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(4), pages 1125-1152.
    28. Tomas Björk & Mariana Khapko & Agatha Murgoci, 2017. "On time-inconsistent stochastic control in continuous time," Finance and Stochastics, Springer, vol. 21(2), pages 331-360, April.
    29. David Laibson, 1997. "Golden Eggs and Hyperbolic Discounting," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 443-478.
    30. Wei He & Yeneng Sun, 2013. "Stationary Markov Perfect Equilibria in Discounted Stochastic Games," Papers 1311.1562, arXiv.org, revised Jan 2017.
    31. Paul A. Samuelson, 1937. "A Note on Measurement of Utility," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 4(2), pages 155-161.
    32. A. S. Nowak & T. E. S. Raghavan, 1992. "Existence of Stationary Correlated Equilibria with Symmetric Information for Discounted Stochastic Games," Mathematics of Operations Research, INFORMS, vol. 17(3), pages 519-526, August.
    33. Charalambos D. Aliprantis & Kim C. Border, 2006. "Infinite Dimensional Analysis," Springer Books, Springer, edition 0, number 978-3-540-29587-7, December.
    34. Łukasz Balbus & Anna Jaśkiewicz & Andrzej S. Nowak, 2015. "Existence of Stationary Markov Perfect Equilibria in Stochastic Altruistic Growth Economies," Journal of Optimization Theory and Applications, Springer, vol. 165(1), pages 295-315, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas J. Sargent & John Stachurski, 2024. "Dynamic Programming: Finite States," Papers 2401.10473, arXiv.org.
    2. Balbus, Łukasz & Reffett, Kevin & Woźny, Łukasz, 2022. "Time-consistent equilibria in dynamic models with recursive payoffs and behavioral discounting," Journal of Economic Theory, Elsevier, vol. 204(C).
    3. Jin-Biao Lu & Zhi-Jiang Liu & Dmitry Tulenty & Liudmila Tsvetkova & Sebastian Kot, 2021. "Implementation of Stochastic Analysis in Corporate Decision-Making Models," Mathematics, MDPI, vol. 9(9), pages 1-16, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lilia Maliar & Serguei Maliar, 2016. "Ruling Out Multiplicity of Smooth Equilibria in Dynamic Games: A Hyperbolic Discounting Example," Dynamic Games and Applications, Springer, vol. 6(2), pages 243-261, June.
    2. Balbus, Łukasz & Reffett, Kevin & Woźny, Łukasz, 2022. "Time-consistent equilibria in dynamic models with recursive payoffs and behavioral discounting," Journal of Economic Theory, Elsevier, vol. 204(C).
    3. Chatterjee, Satyajit & Eyigungor, Burcu, 2016. "Continuous Markov equilibria with quasi-geometric discounting," Journal of Economic Theory, Elsevier, vol. 163(C), pages 467-494.
    4. Jaśkiewicz, Anna & Nowak, Andrzej S., 2014. "Stationary Markov perfect equilibria in risk sensitive stochastic overlapping generations models," Journal of Economic Theory, Elsevier, vol. 151(C), pages 411-447.
    5. Łukasz Balbus & Anna Jaśkiewicz & Andrzej S. Nowak, 2020. "Markov perfect equilibria in a dynamic decision model with quasi-hyperbolic discounting," Annals of Operations Research, Springer, vol. 287(2), pages 573-591, April.
    6. Mariana Khapko, 2023. "Asset pricing with dynamically inconsistent agents," Finance and Stochastics, Springer, vol. 27(4), pages 1017-1046, October.
    7. Balbus, Łukasz & Jaśkiewicz, Anna & Nowak, Andrzej S., 2016. "Non-paternalistic intergenerational altruism revisited," Journal of Mathematical Economics, Elsevier, vol. 63(C), pages 27-33.
    8. Drouhin, Nicolas, 2020. "Non-stationary additive utility and time consistency," Journal of Mathematical Economics, Elsevier, vol. 86(C), pages 1-14.
    9. Luttmer, Erzo G.J. & Mariotti, Thomas, 2007. "Efficiency and equilibrium when preferences are time-inconsistent," Journal of Economic Theory, Elsevier, vol. 132(1), pages 493-506, January.
    10. Jean-Pierre Drugeon & Bertrand Wigniolle, 2021. "On Markovian collective choice with heterogeneous quasi-hyperbolic discounting," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 72(4), pages 1257-1296, November.
    11. Tyson, Christopher J., 2008. "Management of a capital stock by Strotz's naive planner," Journal of Economic Dynamics and Control, Elsevier, vol. 32(7), pages 2214-2239, July.
    12. Camilo Hern'andez & Dylan Possamai, 2020. "Me, myself and I: a general theory of non-Markovian time-inconsistent stochastic control for sophisticated agents," Papers 2002.12572, arXiv.org, revised Jul 2021.
    13. Takeo Hori & Koichi Futagami, 2019. "A Non‐unitary Discount Rate Model," Economica, London School of Economics and Political Science, vol. 86(341), pages 139-165, January.
    14. Lilia Maliar & Serguei Maliar, 2004. "Quasi‐geometric discounting: A closed‐form solution under the exponential utility function," Bulletin of Economic Research, Wiley Blackwell, vol. 56(2), pages 201-206, April.
    15. Kirill Borissov & Mikhail Pakhnin & Ronald Wendner, 2020. "Naive Agents with Quasi-hyperbolic Discounting and Perfect Foresight," EUSP Department of Economics Working Paper Series 2020/03, European University at St. Petersburg, Department of Economics.
    16. Liya Liu & Yingjie Niu & Yuanping Wang & Jinqiang Yang, 2020. "Optimal consumption with time-inconsistent preferences," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 70(3), pages 785-815, October.
    17. Maliar, Lilia & Maliar, Serguei, 2006. "The Neoclassical Growth Model with Heterogeneous Quasi-Geometric Consumers," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(3), pages 635-654, April.
    18. Anke Gerbe & Kirsten I.M. Rohde, 2010. "Risk and Preference Reversals in Intertemporal Choice," Post-Print hal-00911832, HAL.
    19. Balbus, Łukasz & Reffett, Kevin & Woźny, Łukasz, 2018. "On uniqueness of time-consistent Markov policies for quasi-hyperbolic consumers under uncertainty," Journal of Economic Theory, Elsevier, vol. 176(C), pages 293-310.
    20. Gerber, Anke & Rohde, Kirsten I.M., 2010. "Risk and preference reversals in intertemporal choice," Journal of Economic Behavior & Organization, Elsevier, vol. 76(3), pages 654-668, December.

    More about this item

    Keywords

    Markov decision process; Markov perfect equilibrium; Stochastic economic growth;
    All these keywords.

    JEL classification:

    • C61 - Mathematical and Quantitative Methods - - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling - - - Optimization Techniques; Programming Models; Dynamic Analysis
    • C72 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - Noncooperative Games
    • C73 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - Stochastic and Dynamic Games; Evolutionary Games
    • G11 - Financial Economics - - General Financial Markets - - - Portfolio Choice; Investment Decisions

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:finsto:v:25:y:2021:i:2:d:10.1007_s00780-020-00443-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.