IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v317y2022i2d10.1007_s10479-017-2561-9.html
   My bibliography  Save this article

On the optimality equation for average cost Markov decision processes and its validity for inventory control

Author

Listed:
  • Eugene A. Feinberg

    (Stony Brook University)

  • Yan Liang

    (Stony Brook University)

Abstract

As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov decision processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition probabilities and with possibly unbounded one-step costs and noncompact action sets. These conditions also imply the convergence of sequences of discounted relative value functions to average-cost relative value functions and the continuity of average-cost relative value functions. As shown in this paper, the classic periodic-review setup-cost inventory control problem with backorders and convex holding/backlog costs satisfies these conditions. Therefore, the optimality inequality holds in the form of an equality with a continuous average-cost relative value function for this problem. In addition, the K-convexity of discounted relative value functions and their convergence to average-cost relative value functions, when the discount factor increases to 1, imply the K-convexity of average-cost relative value functions. This implies that average-cost optimal (s, S) policies for the inventory control problem can be derived from the average-cost optimality equation.

Suggested Citation

  • Eugene A. Feinberg & Yan Liang, 2022. "On the optimality equation for average cost Markov decision processes and its validity for inventory control," Annals of Operations Research, Springer, vol. 317(2), pages 569-586, October.
    • Handle: RePEc:spr:annopr:v:317:y:2022:i:2:d:10.1007_s10479-017-2561-9
    DOI: 10.1007/s10479-017-2561-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-017-2561-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-017-2561-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Michael Katehakis & Laurens Smit, 2012. "On computing optimal (Q,r) replenishment policies under quantity discounts," Annals of Operations Research, Springer, vol. 200(1), pages 279-298, November.
    2. Eugene A. Feinberg & Mark E. Lewis, 2007. "Optimality Inequalities for Average Cost Markov Decision Processes and the Stochastic Cash Balance Problem," Mathematics of Operations Research, INFORMS, vol. 32(4), pages 769-783, November.
    3. Xin Chen & David Simchi-Levi, 2004. "Coordinating Inventory Control and Pricing Strategies with Random Demand and Fixed Ordering Cost: The Infinite Horizon Case," Mathematics of Operations Research, INFORMS, vol. 29(3), pages 698-723, August.
    4. Junmin Shi & Michael Katehakis & Benjamin Melamed, 2013. "Martingale methods for pricing inventory penalties under continuous replenishment and compound renewal demands," Annals of Operations Research, Springer, vol. 208(1), pages 593-612, September.
    5. Eugene A. Feinberg & Pavlo O. Kasyanov & Nina V. Zadoianchuk, 2012. "Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities," Mathematics of Operations Research, INFORMS, vol. 37(4), pages 591-607, November.
    6. Dirk Beyer & Feng Cheng & Suresh P. Sethi & Michael Taksar, 2010. "Markovian Demand Inventory Models," International Series in Operations Research and Management Science, Springer, number 978-0-387-71604-6, September.
    7. Arthur F. Veinott, Jr. & Harvey M. Wagner, 1965. "Computing Optimal (s, S) Inventory Policies," Management Science, INFORMS, vol. 11(5), pages 525-552, March.
    8. Eugene A. Feinberg & Pavlo O. Kasyanov & Michael Z. Zgurovsky, 2016. "Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities," Mathematics of Operations Research, INFORMS, vol. 41(2), pages 656-681, May.
    9. D. Beyer & S. P. Sethi, 1999. "The Classical Average-Cost Inventory Models of Iglehart and Veinott–Wagner Revisited," Journal of Optimization Theory and Applications, Springer, vol. 101(3), pages 523-555, June.
    10. Xin Chen & David Simchi-Levi, 2004. "Coordinating Inventory Control and Pricing Strategies with Random Demand and Fixed Ordering Cost: The Finite Horizon Case," Operations Research, INFORMS, vol. 52(6), pages 887-896, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eugene A. Feinberg & Yan Liang, 2022. "Structure of optimal policies to periodic-review inventory models with convex costs and backorders for all values of discount factors," Annals of Operations Research, Springer, vol. 317(1), pages 29-45, October.
    2. Eugene A. Feinberg & Mark E. Lewis, 2018. "On the convergence of optimal actions for Markov decision processes and the optimality of (s, S) inventory policies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 65(8), pages 619-637, December.
    3. Wen Chen & Adam J. Fleischhacker & Michael N. Katehakis, 2015. "Dynamic pricing in a dual‐market environment," Naval Research Logistics (NRL), John Wiley & Sons, vol. 62(7), pages 531-549, October.
    4. Qing Li & Xiaoli Wu & Ki Ling Cheung, 2009. "Optimal Policies for Inventory Systems with Separate Delivery-Request and Order-Quantity Decisions," Operations Research, INFORMS, vol. 57(3), pages 626-636, June.
    5. Eugene A. Feinberg & Pavlo O. Kasyanov & Michael Z. Zgurovsky, 2022. "Continuity of equilibria for two-person zero-sum games with noncompact action sets and unbounded payoffs," Annals of Operations Research, Springer, vol. 317(2), pages 537-568, October.
    6. Sandun C. Perera & Suresh P. Sethi, 2023. "A survey of stochastic inventory models with fixed costs: Optimality of (s, S) and (s, S)‐type policies—Discrete‐time case," Production and Operations Management, Production and Operations Management Society, vol. 32(1), pages 131-153, January.
    7. Liu Shuren & Tang Pei, 2014. "The Stochastic Cash Balance Problem with Fixed Costs: The Risk-averse Case," Journal of Systems Science and Information, De Gruyter, vol. 2(6), pages 520-531, December.
    8. Wen Chen & Ying He, 2022. "Dynamic pricing and inventory control with delivery flexibility," Annals of Operations Research, Springer, vol. 317(2), pages 481-508, October.
    9. Ozgun Caliskan-Demirag & Youhua (Frank) Chen & Yi Yang, 2012. "Ordering Policies for Periodic-Review Inventory Systems with Quantity-Dependent Fixed Costs," Operations Research, INFORMS, vol. 60(4), pages 785-796, August.
    10. Vasiliki Kostami, 2020. "Price and Lead time Disclosure Strategies in Inventory Systems," Production and Operations Management, Production and Operations Management Society, vol. 29(12), pages 2760-2788, December.
    11. Walid W. Nasr, 2022. "Inventory systems with stochastic and batch demand: computational approaches," Annals of Operations Research, Springer, vol. 309(1), pages 163-187, February.
    12. Gan, Xianghua & Sethi, Suresh P. & Xu, Liang, 2019. "Simultaneous Optimization of Contingent and Advance Purchase Orders with Fixed Ordering Costs," Omega, Elsevier, vol. 89(C), pages 227-241.
    13. Woonghee Tim Huh & Ganesh Janakiraman & Mahesh Nagarajan, 2011. "Average Cost Single-Stage Inventory Models: An Analysis Using a Vanishing Discount Approach," Operations Research, INFORMS, vol. 59(1), pages 143-155, February.
    14. Ozgun Caliskan-Demirag & Youhua Chen & Yi Yang, 2013. "Production-inventory control policy under warm/cold state-dependent fixed costs and stochastic demand: partial characterization and heuristics," Annals of Operations Research, Springer, vol. 208(1), pages 531-556, September.
    15. M. Güler & Taner Bilgiç & Refik Güllü, 2015. "Joint pricing and inventory control for additive demand models with reference effects," Annals of Operations Research, Springer, vol. 226(1), pages 255-276, March.
    16. Van Foreest, Nicky D. & Kilic, Onur A., 2023. "An intuitive approach to inventory control with optimal stopping," European Journal of Operational Research, Elsevier, vol. 311(3), pages 921-924.
    17. Yonit Barron, 2022. "A probabilistic approach to the stochastic fluid cash management balance problem," Annals of Operations Research, Springer, vol. 312(2), pages 607-645, May.
    18. Hong-Qiao Chen & Xiao-Song Ding & Ji-Hong Zhang & Hua-Yi Li, 2020. "Optimal Production-Inventory Policy for a Periodic-Review Energy Buy-Back System over an Infinite Planning Horizon," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 37(02), pages 1-32, March.
    19. Awi Federgruen & Zhe Liu & Lijian Lu, 2020. "Synthesis and Generalization of Structural Results in Inventory Management: A Generalized Convexity Property," Mathematics of Operations Research, INFORMS, vol. 45(2), pages 547-575, May.
    20. Kaijie Zhu & Ulrich W. Thonemann, 2009. "Coordination of pricing and inventory control across products," Naval Research Logistics (NRL), John Wiley & Sons, vol. 56(2), pages 175-190, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:317:y:2022:i:2:d:10.1007_s10479-017-2561-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.