IDEAS home Printed from https://ideas.repec.org/a/wly/jnljam/v2012y2012i1n103698.html

A Version of the Euler Equation in Discounted Markov Decision Processes

Author

Listed:
  • H. Cruz-Suárez
  • G. Zacarías-Espinoza
  • V. Vázquez-Guevara

Abstract

This paper deals with Markov decision processes (MDPs) on Euclidean spaces with an infinite horizon. An approach to study this kind of MDPs is using the dynamic programming technique (DP). Then the optimal value function is characterized through the value iteration functions. The paper provides conditions that guarantee the convergence of maximizers of the value iteration functions to the optimal policy. Then, using the Euler equation and an envelope formula, the optimal solution of the optimal control problem is obtained. Finally, this theory is applied to a linear‐quadratic control problem in order to find its optimal policy.

Suggested Citation

  • H. Cruz-Suárez & G. Zacarías-Espinoza & V. Vázquez-Guevara, 2012. "A Version of the Euler Equation in Discounted Markov Decision Processes," Journal of Applied Mathematics, John Wiley & Sons, vol. 2012(1).
    • Handle: RePEc:wly:jnljam:v:2012:y:2012:i:1:n:103698
    DOI: 10.1155/2012/103698
    as

    Download full text from publisher

    File URL: https://doi.org/10.1155/2012/103698
    Download Restriction: no
    File URL: https://libkey.io/10.1155/2012/103698?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Fuente,Angel de la, 2000. "Mathematical Methods and Models for Economists," Cambridge Books, Cambridge University Press, number 9780521585293, June.
    2. Benveniste, L M & Scheinkman, J A, 1979. "On the Differentiability of the Value Function in Dynamic Models of Economics," Econometrica, Econometric Society, vol. 47(3), pages 727-732, May.
    3. Mirman, Leonard J. & Zilcha, Itzhak, 1975. "On optimal growth under uncertainty," Journal of Economic Theory, Elsevier, vol. 11(3), pages 329-339, December.
    4. William A. Brock & Leonard J. Mirman, 2001. "Optimal Economic Growth And Uncertainty: The Discounted Case," Chapters, in: W. D. Dechert (ed.), Growth Theory, Nonlinear Dynamics and Economic Modelling, chapter 1, pages 3-37, Edward Elgar Publishing.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Stachurski, 2009. "Economic Dynamics: Theory and Computation," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262012774, December.
    2. Amir, Rabah, 1996. "Sensitivity analysis of multisector optimal economic dynamics," Journal of Mathematical Economics, Elsevier, vol. 25(1), pages 123-141.
    3. Lars J. Olson & Santanu Roy, 2006. "Theory of Stochastic Optimal Economic Growth," Springer Books, in: Rose-Anne Dana & Cuong Le Van & Tapan Mitra & Kazuo Nishimura (ed.), Handbook on Optimal Growth 1, chapter 11, pages 297-335, Springer.
    4. Juan Pablo Rincón-Zapatero, 2020. "Differentiability of the value function and Euler equation in non-concave discrete-time stochastic dynamic programming," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 8(1), pages 79-88, April.
    5. Hugo Cruz-Suárez & Raúl Montes-de-Oca, 2008. "An envelope theorem and some applications to discounted Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 67(2), pages 299-321, April.
    6. Zhang, Yuzhe, 2007. "Stochastic optimal growth with a non-compact state space," Journal of Mathematical Economics, Elsevier, vol. 43(2), pages 115-129, February.
    7. Takashi Kamihigashiw & John Stachurski, 2014. "Seeking Ergodicity in Dynamic Economies," Working Papers 2014-402, Department of Research, Ipag Business School.
    8. Ralph Chami & Thomas F. Cosimano & Connel Fullenkamp, 2001. "Capital Trading, Stock Trading, and the Inflation Tax on Equity," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 4(3), pages 575-606, July.
    9. Tapan Mitra & Santanu Roy, 2023. "Stochastic growth, conservation of capital and convergence to a positive steady state," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 76(1), pages 311-351, July.
    10. Mirman, Leonard J. & Morand, Olivier F. & Reffett, Kevin L., 2008. "A qualitative approach to Markovian equilibrium in infinite horizon economies with capital," Journal of Economic Theory, Elsevier, vol. 139(1), pages 75-98, March.
    11. Gianluca Femminis, 2007. "From simple growth to numerical simulations: a primer in dynamic programming," DISCE - Quaderni dell'Istituto di Teoria Economica e Metodi Quantitativi itemq0745, Università Cattolica del Sacro Cuore, Dipartimenti e Istituti di Scienze Economiche (DISCE).
    12. repec:ipg:wpaper:2014-086 is not listed on IDEAS
    13. Juan Pablo Rinc'on-Zapatero, 2019. "Existence and Uniqueness of Solutions to the Stochastic Bellman Equation with Unbounded Shock," Papers 1907.07343, arXiv.org.
    14. Flam, Sjur Didrik & Mirman, Leonard J., 1998. "Groping for optimal growth," Journal of Economic Dynamics and Control, Elsevier, vol. 23(2), pages 191-207, September.
    15. Mitra, Tapan & Roy, Santanu, 2017. "Optimality of Ramsey–Euler policy in the stochastic growth model," Journal of Economic Theory, Elsevier, vol. 172(C), pages 1-25.
    16. Kazuo Nishimura & Ryszard Rudnicki & John Stachurski, 2012. "Stochastic Optimal Growth with Nonconvexities," Springer Books, in: John Stachurski & Alain Venditti & Makoto Yano (ed.), Nonlinear Dynamics in Equilibrium Models, edition 127, chapter 0, pages 261-288, Springer.
    17. Manjira Datta & Leonard Mirman & Olivier Morand & Kevin Reffett, 2002. "Monotone Methods for Markovian Equilibrium in Dynamic Economies," Annals of Operations Research, Springer, vol. 114(1), pages 117-144, August.
    18. Mitra, Tapan & Roy, Santanu, 2012. "Sustained positive consumption in a model of stochastic growth: The role of risk aversion," Journal of Economic Theory, Elsevier, vol. 147(2), pages 850-880.
    19. Giuseppe Di Vita, 2023. "The economic impact of legislative complexity and corruption: A cross‐country analysis," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 28(2), pages 1801-1825, April.
    20. Kazuo Nishimura & John Stachurski, 2012. "Stability of Stochastic Optimal Growth Models: A New Approach," Springer Books, in: John Stachurski & Alain Venditti & Makoto Yano (ed.), Nonlinear Dynamics in Equilibrium Models, edition 127, chapter 0, pages 289-307, Springer.
    21. Mitra, Tapan & Privileggi, Fabio, 2006. "Cantor type attractors in stochastic growth models," Chaos, Solitons & Fractals, Elsevier, vol. 29(3), pages 626-637.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:jnljam:v:2012:y:2012:i:1:n:103698. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://onlinelibrary.wiley.com/journal/4185 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.