IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v296y2022i3p953-966.html
   My bibliography  Save this article

Markov decision processes with recursive risk measures

Author

Listed:
  • Bäuerle, Nicole
  • Glauner, Alexander

Abstract

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost. We treat both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature. It has been studied before for the entropic risk measure and is extended here to general static risk measures. Under direct assumptions on the model data we derive a Bellman equation and prove the existence of optimal Markov policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Our approach unifies results for a number of well-known risk measures. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.

Suggested Citation

  • Bäuerle, Nicole & Glauner, Alexander, 2022. "Markov decision processes with recursive risk measures," European Journal of Operational Research, Elsevier, vol. 296(3), pages 953-966.
    • Handle: RePEc:eee:ejores:v:296:y:2022:i:3:p:953-966
    DOI: 10.1016/j.ejor.2021.04.030
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S037722172100360X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2021.04.030?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Homem-de-Mello, Tito & Pagnoncelli, Bernardo K., 2016. "Risk aversion in multistage stochastic programming: A modeling and algorithmic perspective," European Journal of Operational Research, Elsevier, vol. 249(1), pages 188-199.
    2. Shapiro, Alexander & Tekaya, Wajdi & da Costa, Joari Paulo & Soares, Murilo Pereira, 2013. "Risk neutral and risk averse Stochastic Dual Dynamic Programming method," European Journal of Operational Research, Elsevier, vol. 224(2), pages 375-391.
    3. Epstein, Larry G. & Schneider, Martin, 2003. "Recursive multiple-priors," Journal of Economic Theory, Elsevier, vol. 113(1), pages 1-31, November.
    4. Kai Detlefsen & Giacomo Scandolo, 2005. "Conditional and Dynamic Convex Risk Measures," SFB 649 Discussion Papers SFB649DP2005-006, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    5. Rudloff, Birgit & Street, Alexandre & Valladão, Davi M., 2014. "Time consistency and risk averse dynamic decision models: Definition, interpretation and practical consequences," European Journal of Operational Research, Elsevier, vol. 234(3), pages 743-750.
    6. Bion-Nadal, Jocelyne, 2009. "Time consistent dynamic risk processes," Stochastic Processes and their Applications, Elsevier, vol. 119(2), pages 633-654, February.
    7. Andy Philpott & Vitor de Matos & Erlon Finardi, 2013. "On Solving Multistage Stochastic Programs with Coherent Risk Measures," Operations Research, INFORMS, vol. 61(4), pages 957-970, August.
    8. Riedel, Frank, 2004. "Dynamic coherent risk measures," Stochastic Processes and their Applications, Elsevier, vol. 112(2), pages 185-200, August.
    9. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2018. "A Unified Approach to Time Consistency of Dynamic Risk Measures and Dynamic Performance Measures in Discrete Time," Mathematics of Operations Research, INFORMS, vol. 43(1), pages 204-221, February.
    10. Nicole Bäuerle & Jonathan Ott, 2011. "Markov Decision Processes with Average-Value-at-Risk criteria," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 74(3), pages 361-379, December.
    11. Schur, Rouven & Gönsch, Jochen & Hassler, Michael, 2019. "Time-consistent, risk-averse dynamic pricing," European Journal of Operational Research, Elsevier, vol. 277(2), pages 587-603.
    12. Ronald A. Howard & James E. Matheson, 1972. "Risk-Sensitive Markov Decision Processes," Management Science, INFORMS, vol. 18(7), pages 356-369, March.
    13. Bäuerle, Nicole & Jaśkiewicz, Anna, 2017. "Optimal dividend payout model with risk sensitive preferences," Insurance: Mathematics and Economics, Elsevier, vol. 73(C), pages 82-93.
    14. Kai Detlefsen & Giacomo Scandolo, 2005. "Conditional and dynamic convex risk measures," Finance and Stochastics, Springer, vol. 9(4), pages 539-561, October.
    15. Stefan Weber, 2006. "Distribution‐Invariant Risk Measures, Information, And Dynamic Consistency," Mathematical Finance, Wiley Blackwell, vol. 16(2), pages 419-441, April.
    16. Bäuerle, Nicole & Jaśkiewicz, Anna, 2018. "Stochastic optimal growth model with risk sensitive preferences," Journal of Economic Theory, Elsevier, vol. 173(C), pages 181-200.
    17. Bäuerle, Nicole & Glauner, Alexander, 2018. "Optimal risk allocation in reinsurance networks," Insurance: Mathematics and Economics, Elsevier, vol. 82(C), pages 37-47.
    18. Kreps, David M & Porteus, Evan L, 1978. "Temporal Resolution of Uncertainty and Dynamic Choice Theory," Econometrica, Econometric Society, vol. 46(1), pages 185-200, January.
    19. Pichler, Alois, 2013. "The natural Banach space for version independent risk measures," Insurance: Mathematics and Economics, Elsevier, vol. 53(2), pages 405-415.
    20. Charalambos D. Aliprantis & Kim C. Border, 2006. "Infinite Dimensional Analysis," Springer Books, Springer, edition 0, number 978-3-540-29587-7, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas J. Sargent & John Stachurski, 2024. "Dynamic Programming: Finite States," Papers 2401.10473, arXiv.org.
    2. Zheng, Rui & Wang, Jingjing & Zhang, Yingzhi, 2023. "A hybrid repair-replacement policy in the proportional hazards model," European Journal of Operational Research, Elsevier, vol. 304(3), pages 1011-1021.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicole Bauerle & Alexander Glauner, 2020. "Markov Decision Processes with Recursive Risk Measures," Papers 2010.07220, arXiv.org.
    2. Bellini, Fabio & Laeven, Roger J.A. & Rosazza Gianin, Emanuela, 2021. "Dynamic robust Orlicz premia and Haezendonck–Goovaerts risk measures," European Journal of Operational Research, Elsevier, vol. 291(2), pages 438-446.
    3. Cheridito, Patrick & Stadje, Mitja, 2009. "Time-inconsistency of VaR and time-consistent alternatives," Finance Research Letters, Elsevier, vol. 6(1), pages 40-46, March.
    4. Dan A. Iancu & Marek Petrik & Dharmashankar Subramanian, 2015. "Tight Approximations of Dynamic Risk Measures," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 655-682, March.
    5. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2014. "A unified approach to time consistency of dynamic risk measures and dynamic performance measures in discrete time," Papers 1409.7028, arXiv.org, revised Sep 2017.
    6. Rudloff, Birgit & Street, Alexandre & Valladão, Davi M., 2014. "Time consistency and risk averse dynamic decision models: Definition, interpretation and practical consequences," European Journal of Operational Research, Elsevier, vol. 234(3), pages 743-750.
    7. De Lara, Michel & Leclère, Vincent, 2016. "Building up time-consistency for risk measures and dynamic optimization," European Journal of Operational Research, Elsevier, vol. 249(1), pages 177-187.
    8. Schur, Rouven & Gönsch, Jochen & Hassler, Michael, 2019. "Time-consistent, risk-averse dynamic pricing," European Journal of Operational Research, Elsevier, vol. 277(2), pages 587-603.
    9. Henri Gérard & Michel Lara & Jean-Philippe Chancelier, 2020. "Equivalence between time consistency and nested formula," Annals of Operations Research, Springer, vol. 292(2), pages 627-647, September.
    10. Lacker Daniel, 2018. "Law invariant risk measures and information divergences," Dependence Modeling, De Gruyter, vol. 6(1), pages 228-258, November.
    11. D. Madan & M. Pistorius & M. Stadje, 2017. "On dynamic spectral risk measures, a limit theorem and optimal portfolio allocation," Finance and Stochastics, Springer, vol. 21(4), pages 1073-1102, October.
    12. Zachary Feinstein & Birgit Rudloff, 2018. "Scalar multivariate risk measures with a single eligible asset," Papers 1807.10694, arXiv.org, revised Feb 2021.
    13. Qinyu Wu & Fan Yang & Ping Zhang, 2023. "Conditional generalized quantiles based on expected utility model and equivalent characterization of properties," Papers 2301.12420, arXiv.org.
    14. Acciaio, Beatrice & Föllmer, Hans & Penner, Irina, 2012. "Risk assessment for uncertain cash flows: model ambiguity, discounting ambiguity, and the role of bubbles," LSE Research Online Documents on Economics 50118, London School of Economics and Political Science, LSE Library.
    15. Nicole EL KAROUI & Claudia RAVANELLI, 2008. "Cash Sub-additive Risk Measures and Interest Rate Ambiguity," Swiss Finance Institute Research Paper Series 08-09, Swiss Finance Institute.
    16. Zachary Feinstein & Birgit Rudloff, 2018. "Time consistency for scalar multivariate risk measures," Papers 1810.04978, arXiv.org, revised Nov 2021.
    17. Zachary Feinstein & Birgit Rudloff, 2015. "Multi-portfolio time consistency for set-valued convex and coherent risk measures," Finance and Stochastics, Springer, vol. 19(1), pages 67-107, January.
    18. Alexander S. Cherny, 2009. "Capital Allocation And Risk Contribution With Discrete‐Time Coherent Risk," Mathematical Finance, Wiley Blackwell, vol. 19(1), pages 13-40, January.
    19. Zhiping Chen & Jia Liu & Gang Li & Zhe Yan, 2016. "Composite time-consistent multi-period risk measure and its application in optimal portfolio selection," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 24(3), pages 515-540, October.
    20. Andreas H Hamel, 2018. "Monetary Measures of Risk," Papers 1812.04354, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:296:y:2022:i:3:p:953-966. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.