IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v50y2002i5p796-809.html
   My bibliography  Save this article

Structural Properties of Stochastic Dynamic Programs

Author

Listed:
  • James E. Smith

    (Fuqua School of Business, Duke University, Box 90120, Durham, North Carolina 27708-0120)

  • Kevin F. McCardle

    (The Anderson School at UCLA, 110 Westwood Plaza, Suite D520, Los Angeles, California 90095-1481)

Abstract

In Markov models of sequential decision processes, one is often interested in showing that the value function is monotonic, convex, and/or supermodular in the state variables. These kinds of results can be used to develop a qualitative understanding of the model and characterize how the results will change with changes in model parameters. In this paper we present several fundamental results for establishing these kinds of properties. The results are, in essence, "metatheorems" showing that the value functions satisfy property P if the reward functions satisfy property P and the transition probabilities satisfy a stochastic version of this property. We focus our attention on closed convex cone properties, a large class of properties that includes monotonicity, convexity, and supermodularity, as well as combinations of these and many other properties of interest.

Suggested Citation

  • James E. Smith & Kevin F. McCardle, 2002. "Structural Properties of Stochastic Dynamic Programs," Operations Research, INFORMS, vol. 50(5), pages 796-809, October.
    • Handle: RePEc:inm:oropre:v:50:y:2002:i:5:p:796-809
    DOI: 10.1287/opre.50.5.796.365
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/opre.50.5.796.365
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.50.5.796.365?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Samuel Karlin, 1960. "Dynamic Inventory Policy with Varying Stochastic Demands," Management Science, INFORMS, vol. 6(3), pages 231-258, April.
    2. Steven A. Lippman & Kevin F. McCardle, 1987. "Does Cheaper, Faster, or Better Imply Sooner in the Timing of Innovation Decisions?," Management Science, INFORMS, vol. 33(8), pages 1058-1064, August.
    3. Brennan, Michael J & Schwartz, Eduardo S, 1985. "Evaluating Natural Resource Investments," The Journal of Business, University of Chicago Press, vol. 58(2), pages 135-157, April.
    4. Kevin F. McCardle, 1985. "Information Acquisition and the Adoption of New Technology," Management Science, INFORMS, vol. 31(11), pages 1372-1389, November.
    5. Milgrom, Paul & Shannon, Chris, 1994. "Monotone Comparative Statics," Econometrica, Econometric Society, vol. 62(1), pages 157-180, January.
    6. Robert C. Merton, 2005. "Theory of rational option pricing," World Scientific Book Chapters, in: Sudipto Bhattacharya & George M Constantinides (ed.), Theory Of Valuation, chapter 8, pages 229-288, World Scientific Publishing Co. Pte. Ltd..
    7. Rothschild, Michael & Stiglitz, Joseph E., 1970. "Increasing risk: I. A definition," Journal of Economic Theory, Elsevier, vol. 2(3), pages 225-243, September.
    8. Cox, John C. & Ross, Stephen A. & Rubinstein, Mark, 1979. "Option pricing: A simplified approach," Journal of Financial Economics, Elsevier, vol. 7(3), pages 229-263, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Victor F. Araman & Ioana Popescu, 2010. "Media Revenue Management with Audience Uncertainty: Balancing Upfront and Spot Market Sales," Manufacturing & Service Operations Management, INFORMS, vol. 12(2), pages 190-212, December.
    2. Ruitu Xu & Yifei Min & Tianhao Wang & Zhaoran Wang & Michael I. Jordan & Zhuoran Yang, 2023. "Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning," Papers 2303.04833, arXiv.org.
    3. Alessandro Arlotto & J. Michael Steele, 2018. "A Central Limit Theorem for Costs in Bulinskaya’s Inventory Management Problem When Deliveries Face Delays," Methodology and Computing in Applied Probability, Springer, vol. 20(3), pages 839-854, September.
    4. Zhuang, Weifen & Li, Michael Z.F., 2012. "Monotone optimal control for a class of Markov decision processes," European Journal of Operational Research, Elsevier, vol. 217(2), pages 342-350.
    5. Vits, Jeroen & Gelders, Ludo & Pintelon, Liliane, 2006. "Production process changes: A dynamic programming approach to manage effective capacity and experience," International Journal of Production Economics, Elsevier, vol. 104(2), pages 473-481, December.
    6. Anton Ovchinnikov & Béatrice Boulu-Reshef & Phillip E. Pfeifer, 2014. "Balancing Acquisition and Retention Spending for Firms with Limited Capacity," Management Science, INFORMS, vol. 60(8), pages 2002-2019, August.
    7. Paul Zipkin, 2008. "On the Structure of Lost-Sales Inventory Models," Operations Research, INFORMS, vol. 56(4), pages 937-944, August.
    8. James E. Smith & Canan Ulu, 2017. "Risk Aversion, Information Acquisition, and Technology Adoption," Operations Research, INFORMS, vol. 65(4), pages 1011-1028, August.
    9. Borrero, J.S. & Akhavan-Tabatabaei, R., 2013. "Time and inventory dependent optimal maintenance policies for single machine workstations: An MDP approach," European Journal of Operational Research, Elsevier, vol. 228(3), pages 545-555.
    10. Bar Light, 2021. "Stochastic Comparative Statics in Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 46(2), pages 797-810, May.
    11. Saghafian, Soroush, 2018. "Ambiguous partially observable Markov decision processes: Structural results and applications," Journal of Economic Theory, Elsevier, vol. 178(C), pages 1-35.
    12. Simone Cerreia-Vioglio & Fabio Maccheroni & Massimo Marinacci, 2017. "Stochastic Dominance Analysis Without the Independence Axiom," Management Science, INFORMS, vol. 63(4), pages 1097-1109, April.
    13. Emre Nadar & Mustafa Akan & Alan Scheller-Wolf, 2014. "Technical Note---Optimal Structural Results for Assemble-to-Order Generalized M -Systems," Operations Research, INFORMS, vol. 62(3), pages 571-579, June.
    14. Ioana Popescu & Yaozhong Wu, 2007. "Dynamic Pricing Strategies with Reference Effects," Operations Research, INFORMS, vol. 55(3), pages 413-429, June.
    15. Samuel Vercraene & Jean-Philippe Gayon & Fikri Karaesmen, 2018. "Effects of System Parameters on the Optimal Cost and Policy in a Class of Multidimensional Queueing Control Problems," Operations Research, INFORMS, vol. 66(1), pages 150-162, January.
    16. Sachin Adlakha & Ramesh Johari, 2013. "Mean Field Equilibrium in Dynamic Games with Strategic Complementarities," Operations Research, INFORMS, vol. 61(4), pages 971-989, August.
    17. Wang, Jun & Zhu, Xiaoyan, 2021. "Joint optimization of condition-based maintenance and inventory control for a k-out-of-n:F system of multi-state degrading components," European Journal of Operational Research, Elsevier, vol. 290(2), pages 514-529.
    18. Ayc{s}e Kocab{i}y{i}kou{g}lu & Ioana Popescu, 2007. "Managerial Motivation Dynamics and Incentives," Management Science, INFORMS, vol. 53(5), pages 834-848, May.
    19. Kiran Krishnamurthy, Chandra, 2012. "Optimal Management of Groundwater under Uncertainty: A Unified Approach," CERE Working Papers 2012:19, CERE - the Center for Environmental and Resource Economics, revised 30 Jun 2014.
    20. Daniel R. Jiang & Warren B. Powell, 2015. "An Approximate Dynamic Programming Algorithm for Monotone Value Functions," Operations Research, INFORMS, vol. 63(6), pages 1489-1511, December.
    21. James E. Smith & Canan Ulu, 2012. "Technology Adoption with Uncertain Future Costs and Quality," Operations Research, INFORMS, vol. 60(2), pages 262-274, April.
    22. Bar Light, 2019. "Stochastic Comparative Statics in Markov Decision Processes," Papers 1904.05481, arXiv.org, revised Jan 2020.
    23. Sheldon M. Ross & Gideon Weiss & Zhengyu Zhang, 2021. "Technical Note—A Stochastic Assignment Problem with Unknown Eligibility Probabilities," Operations Research, INFORMS, vol. 69(1), pages 266-272, January.
    24. Lauren E. Cipriano & Thomas A. Weber, 2018. "Population-level intervention and information collection in dynamic healthcare policy," Health Care Management Science, Springer, vol. 21(4), pages 604-631, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. José Balibrea-Iniesta, 2020. "Economic Analysis of Renewable Energy Regulation in France: A Case Study for Photovoltaic Plants Based on Real Options," Energies, MDPI, vol. 13(11), pages 1-19, June.
    2. Madlener, Reinhard & Stoverink, Simon, 2012. "Power plant investments in the Turkish electricity sector: A real options approach taking into account market liberalization," Applied Energy, Elsevier, vol. 97(C), pages 124-134.
    3. Suresh M. Sundaresan, 2000. "Continuous‐Time Methods in Finance: A Review and an Assessment," Journal of Finance, American Finance Association, vol. 55(4), pages 1569-1622, August.
    4. Marcel Philipp Müller & Sebastian Stöckl & Steffen Zimmermann & Bernd Heinrich, 2016. "Decision Support for IT Investment Projects," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 58(6), pages 381-396, December.
    5. Barbara Glensk & Reinhard Madlener, 2019. "Energiewende @ Risk: On the Continuation of Renewable Power Generation at the End of Public Policy Support," Energies, MDPI, vol. 12(19), pages 1-25, September.
    6. Chung-Gee Lin & Yu-Shan Wang, 2012. "Evaluating natural resource projects with embedded options and limited reserves," Applied Economics, Taylor & Francis Journals, vol. 44(12), pages 1471-1482, April.
    7. Miranda, Oscar & Brandão, Luiz E. & Lazo Lazo, Juan, 2017. "A dynamic model for valuing flexible mining exploration projects under uncertainty," Resources Policy, Elsevier, vol. 52(C), pages 393-404.
    8. repec:dau:papers:123456789/1046 is not listed on IDEAS
    9. Robert Kast & André Lapied & Sophie Pardo & Camelia Protopopescu, 2001. "Évaluation de risques controversés par la théorie des options réelles," Economie & Prévision, La Documentation Française, vol. 149(3), pages 51-63.
    10. Dimson, Elroy & Mussavian, Massoud, 1999. "Three centuries of asset pricing," Journal of Banking & Finance, Elsevier, vol. 23(12), pages 1745-1769, December.
    11. James E. Smith & Kevin F. McCardle, 1999. "Options in the Real World: Lessons Learned in Evaluating Oil and Gas Investments," Operations Research, INFORMS, vol. 47(1), pages 1-15, February.
    12. Donald Brown & Rustam Ibragimov, 2005. "Sign Tests for Dependent Observations and Bounds for Path-Dependent Options," Yale School of Management Working Papers amz2581, Yale School of Management, revised 01 Jul 2005.
    13. Zhang, Mingming & Zhou, Dequn & Zhou, Peng, 2014. "A real option model for renewable energy policy evaluation with application to solar PV power generation in China," Renewable and Sustainable Energy Reviews, Elsevier, vol. 40(C), pages 944-955.
    14. Dalbem, Marta Corrêa & Brandão, Luiz Eduardo Teixeira & Gomes, Leonardo Lima, 2014. "Can the regulated market help foster a free market for wind energy in Brazil?," Energy Policy, Elsevier, vol. 66(C), pages 303-311.
    15. Akamatsu, Takashi & Nagae, Takeshi, 2011. "A network of options: Evaluating complex interdependent decisions under uncertainty," Journal of Economic Dynamics and Control, Elsevier, vol. 35(5), pages 714-729, May.
    16. Schachter, J.A. & Mancarella, P., 2016. "A critical review of Real Options thinking for valuing investment flexibility in Smart Grids and low carbon energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 261-271.
    17. Perrakis, Stylianos, 1989. "Les contributions de la théorie financière à la solution de problèmes en organisation industrielle et en microéconomie appliquée," L'Actualité Economique, Société Canadienne de Science Economique, vol. 65(4), pages 518-546, décembre.
    18. Canan Ulu & James E. Smith, 2009. "Uncertainty, Information Acquisition, and Technology Adoption," Operations Research, INFORMS, vol. 57(3), pages 740-752, June.
    19. Christophe Boucher, 2003. "La valorisation des sociétés de la Nouvelle économie par les options réelles : vertiges et controverses d’une analogie," Revue d'Économie Financière, Programme National Persée, vol. 72(3), pages 299-315.
    20. Jack Clark Francis & Arie Harel & Giora Harpaz, 2010. "Actuarially Fair Premia for Deductible Insurance Policies," The American Economist, Sage Publications, vol. 55(2), pages 83-91, November.
    21. Carlos Andrés Zapata Quimbayo, 2020. "OPCIONES REALES Una guía teórico-práctica para la valoración de inversiones bajo incertidumbre mediante modelos en tiempo discreto y simulación de Monte Carlo," Books, Universidad Externado de Colombia, Facultad de Finanzas, Gobierno y Relaciones Internacionales, number 138, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:50:y:2002:i:5:p:796-809. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.