IDEAS home Printed from https://ideas.repec.org/a/spr/mathme/v67y2008i2p299-321.html

An envelope theorem and some applications to discounted Markov decision processes

Author

Listed:
  • Hugo Cruz-Suárez

  • Raúl Montes-de-Oca

Abstract

In this paper, an Envelope Theorem (ET) will be established for optimization problems on Euclidean spaces. In general, the Envelope Theorems permit analyzing an optimization problem and giving the solution by means of differentiability techniques. The ET will be presented in two versions. One of them uses concavity assumptions, whereas the other one does not require such kind of assumptions. Thereafter, the ET established will be applied to the Markov Decision Processes (MDPs) on Euclidean spaces, discounted and with infinite horizon. As the first application, several examples (including some economic models) of discounted MDPs for which the et allows to determine the value iteration functions will be presented. This will permit to obtain the corresponding optimal value functions and the optimal policies. As the second application of the ET, it will be proved that under differentiability conditions in the transition law, in the reward function, and the noise of the system, the value function and the optimal policy of the problem are differentiable with respect to the state of the system. Besides, various examples to illustrate these differentiability conditions will be provided. Copyright Springer-Verlag 2008

Suggested Citation

  • Hugo Cruz-Suárez & Raúl Montes-de-Oca, 2008. "An envelope theorem and some applications to discounted Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 67(2), pages 299-321, April.
    • Handle: RePEc:spr:mathme:v:67:y:2008:i:2:p:299-321
    DOI: 10.1007/s00186-007-0155-z
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s00186-007-0155-z
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s00186-007-0155-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Paul Milgrom & Ilya Segal, 2002. "Envelope Theorems for Arbitrary Choice Sets," Econometrica, Econometric Society, vol. 70(2), pages 583-601, March.
    2. Fuente,Angel de la, 2000. "Mathematical Methods and Models for Economists," Cambridge Books, Cambridge University Press, number 9780521585293, January.
    3. Benveniste, L M & Scheinkman, J A, 1979. "On the Differentiability of the Value Function in Dynamic Models of Economics," Econometrica, Econometric Society, vol. 47(3), pages 727-732, May.
    4. Daniel Cruz-Suárez & Raúl Montes-de-Oca & Francisco Salem-Silva, 2004. "Conditions for the uniqueness of optimal policies of discounted Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 60(3), pages 415-436, December.
    5. Santos, Manuel S., 1999. "Numerical solution of dynamic economic models," Handbook of Macroeconomics, in: J. B. Taylor & M. Woodford (ed.), Handbook of Macroeconomics, edition 1, volume 1, chapter 5, pages 311-386, Elsevier.
    6. William A. Brock & Leonard J. Mirman, 2001. "Optimal Economic Growth And Uncertainty: The Discounted Case," Chapters, in: W. D. Dechert (ed.), Growth Theory, Nonlinear Dynamics and Economic Modelling, chapter 1, pages 3-37, Edward Elgar Publishing.
    7. Santos, Manuel S., 1994. "Smooth dynamics and computation in models of economic growth," Journal of Economic Dynamics and Control, Elsevier, vol. 18(3-4), pages 879-895.
    8. Jerusalem D. Levhari & T. N. Srinivasan, 1969. "Optimal Savings under Uncertainty," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 36(2), pages 153-163.
    9. Amir, Rabah, 1997. "A new look at optimal growth under uncertainty," Journal of Economic Dynamics and Control, Elsevier, vol. 22(1), pages 67-86, November.
    10. Blume, Lawrence & Easley, David & O'Hara, Maureen, 1982. "Characterization of optimal plans for stochastic dynamic programs," Journal of Economic Theory, Elsevier, vol. 28(2), pages 221-234, December.
    11. Araujo, A & Scheinkman, Jose A, 1977. "Smoothness, Comparative Dynamics, and the Turnpike Property," Econometrica, Econometric Society, vol. 45(3), pages 601-620, April.
    12. Marvin Kraus, 2002. "A generalized envelope theorem with an application to congestion-prone facilities," Economics Bulletin, AccessEcon, vol. 3(28), pages 1-4.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gladys Denisse Salgado Su¨¢rez & Hugo Cruz-Su¨¢rez & Jos¨¦ Dionicio Zacar¨ªas Flores, 2018. "Asymptotic Analysis of a Deterministic Control System via Euler's Equation Approach," Journal of Mathematics Research, Canadian Center of Science and Education, vol. 10(1), pages 115-123, February.
    2. Mauro Gaggero & Giorgio Gnecco & Marcello Sanguineti, 2014. "Approximate dynamic programming for stochastic N-stage optimization with application to optimal consumption under uncertainty," Computational Optimization and Applications, Springer, vol. 58(1), pages 31-85, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lars J. Olson & Santanu Roy, 2006. "Theory of Stochastic Optimal Economic Growth," Springer Books, in: Rose-Anne Dana & Cuong Le Van & Tapan Mitra & Kazuo Nishimura (ed.), Handbook on Optimal Growth 1, chapter 11, pages 297-335, Springer.
    2. John Stachurski, 2009. "Economic Dynamics: Theory and Computation," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262012774, December.
    3. Juan Pablo Rincón-Zapatero, 2020. "Differentiability of the value function and Euler equation in non-concave discrete-time stochastic dynamic programming," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 8(1), pages 79-88, April.
    4. Amir, Rabah, 1996. "Sensitivity analysis of multisector optimal economic dynamics," Journal of Mathematical Economics, Elsevier, vol. 25(1), pages 123-141.
    5. Chen, Yu & Cosimano, Thomas F. & Himonas, Alex A., 2008. "Analytic solving of asset pricing models: The by force of habit case," Journal of Economic Dynamics and Control, Elsevier, vol. 32(11), pages 3631-3660, November.
    6. Coleman, Wilbur John, II, 1991. "Equilibrium in a Production Economy with an Income Tax," Econometrica, Econometric Society, vol. 59(4), pages 1091-1104, July.
    7. Santos, Manuel S., 2004. "Simulation-based estimation of dynamic models with continuous equilibrium solutions," Journal of Mathematical Economics, Elsevier, vol. 40(3-4), pages 465-491, June.
    8. Williams, Noah, 2004. "Small noise asymptotics for a stochastic growth model," Journal of Economic Theory, Elsevier, vol. 119(2), pages 271-298, December.
    9. Clausen, Andrew & Strub, Carlo, 2020. "Reverse Calculus and nested optimization," Journal of Economic Theory, Elsevier, vol. 187(C).
    10. Andrew Clausen & Carlo Strub, 2012. "Envelope theorems for non-smooth and non-concave optimization," ECON - Working Papers 062, Department of Economics - University of Zurich.
    11. Santos, Manuel S., 2003. "Simulation-based estimation of dynamic models with continuous equilibrium solutions," UC3M Working papers. Economics we034716, Universidad Carlos III de Madrid. Departamento de Economía.
    12. Aliprantis, C.D. & Camera, G. & Ruscitti, F., 2007. "Monetary Equilibrium and the Differentiability of the Value Function," Purdue University Economics Working Papers 1199, Purdue University, Department of Economics.
    13. Timothy J. Kehoe & David K. Levine & Paul Romer, 1989. "Steady States and Determinacy in Economies with Infinitely Lived Agents," Levine's Working Paper Archive 52, David K. Levine.
    14. Aliprantis, C.D. & Camera, G. & Ruscitti, F., 2009. "Monetary equilibrium and the differentiability of the value function," Journal of Economic Dynamics and Control, Elsevier, vol. 33(2), pages 454-462, February.
    15. Kazuo Nishimura & Ryszard Rudnicki & John Stachurski, 2004. "Stochastic Growth With Nonconvexities:The Optimal Case," Department of Economics - Working Papers Series 897, The University of Melbourne.
    16. Cuong Le Van & Lisa Morhaim, 2006. "On optimal growth models when the discount factor is near 1 or equal to 1," Post-Print halshs-00096034, HAL.
    17. Lilia Maliar & Serguei Maliar & John B. Taylor & Inna Tsener, 2020. "A tractable framework for analyzing a class of nonstationary Markov models," Quantitative Economics, Econometric Society, vol. 11(4), pages 1289-1323, November.
    18. Tapan Mitra & Santanu Roy, 2023. "Stochastic growth, conservation of capital and convergence to a positive steady state," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 76(1), pages 311-351, July.
    19. Mirman, Leonard J. & Morand, Olivier F. & Reffett, Kevin L., 2008. "A qualitative approach to Markovian equilibrium in infinite horizon economies with capital," Journal of Economic Theory, Elsevier, vol. 139(1), pages 75-98, March.
    20. de Castro, Luciano I. & Galvao, Antonio F. & Nunes, Daniel da Siva, 2025. "Dynamic economics with quantile preferences," Theoretical Economics, Econometric Society, vol. 20(1), January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:mathme:v:67:y:2008:i:2:p:299-321. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.