IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v199y2012i1p193-21410.1007-s10479-012-1070-0.html
   My bibliography  Save this article

Illustrated review of convergence conditions of the value iteration algorithm and the rolling horizon procedure for average-cost MDPs

Author

Listed:
  • Eugenio Vecchia
  • Silvia Marco
  • Alain Jean-Marie

Abstract

This paper is concerned with the links between the Value Iteration algorithm and the Rolling Horizon procedure, for solving problems of stochastic optimal control under the long-run average criterion, in Markov Decision Processes with finite state and action spaces. We review conditions of the literature which imply the geometric convergence of Value Iteration to the optimal value. Aperiodicity is an essential prerequisite for convergence. We prove that the convergence of Value Iteration generally implies that of Rolling Horizon. We also present a modified Rolling Horizon procedure that can be applied to models without analyzing periodicity, and discuss the impact of this transformation on convergence. We illustrate with numerous examples the different results. Finally, we discuss rules for stopping Value Iteration or finding the length of a Rolling Horizon. We provide an example which demonstrates the difficulty of the question, disproving in particular a conjectured rule proposed by Puterman. Copyright Springer Science+Business Media, LLC 2012

Suggested Citation

  • Eugenio Vecchia & Silvia Marco & Alain Jean-Marie, 2012. "Illustrated review of convergence conditions of the value iteration algorithm and the rolling horizon procedure for average-cost MDPs," Annals of Operations Research, Springer, vol. 199(1), pages 193-214, October.
    • Handle: RePEc:spr:annopr:v:199:y:2012:i:1:p:193-214:10.1007/s10479-012-1070-0
    DOI: 10.1007/s10479-012-1070-0
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10479-012-1070-0
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s10479-012-1070-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jeffrey M. Alden & Robert L. Smith, 1992. "Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes," Operations Research, INFORMS, vol. 40(3-supplem), pages 183-194, June.
    2. P. J. Schweitzer & A. Federgruen, 1977. "The Asymptotic Behavior of Undiscounted Value Iteration in Markov Decision Problems," Mathematics of Operations Research, INFORMS, vol. 2(4), pages 360-381, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Seksan Kiatsupaibul & Robert L. Smith & Zelda B. Zabinsky, 2016. "Solving infinite horizon optimization problems through analysis of a one-dimensional global optimization problem," Journal of Global Optimization, Springer, vol. 66(4), pages 711-727, December.
    2. Torpong Cheevaprawatdomrong & Robert L. Smith, 2004. "Infinite Horizon Production Scheduling in Time-Varying Systems Under Stochastic Demand," Operations Research, INFORMS, vol. 52(1), pages 105-115, February.
    3. Allise O. Wachs & Irwin E. Schochetman & Robert L. Smith, 2011. "Average Optimality in Nonhomogeneous Infinite Horizon Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 36(1), pages 147-164, February.
    4. Chevalier, Philippe & Lamas, Alejandro & Lu, Liang & Mlinar, Tanja, 2015. "Revenue management for operations with urgent orders," European Journal of Operational Research, Elsevier, vol. 240(2), pages 476-487.
    5. Sinitskaya, Ekaterina & Tesfatsion, Leigh, 2015. "Macroeconomies as constructively rational games," Journal of Economic Dynamics and Control, Elsevier, vol. 61(C), pages 152-182.
    6. Martinelli, Gabriele & Eidsvik, Jo & Hauge, Ragnar, 2013. "Dynamic decision making for graphical models applied to oil exploration," European Journal of Operational Research, Elsevier, vol. 230(3), pages 688-702.
    7. Esmaeil Keyvanshokooh & Pooyan Kazemian & Mohammad Fattahi & Mark P. Van Oyen, 2022. "Coordinated and Priority‐Based Surgical Care: An Integrated Distributionally Robust Stochastic Optimization Approach," Production and Operations Management, Production and Operations Management Society, vol. 31(4), pages 1510-1535, April.
    8. Irina S. Dolinskaya, 2012. "Optimal path finding in direction, location, and time dependent environments," Naval Research Logistics (NRL), John Wiley & Sons, vol. 59(5), pages 325-339, August.
    9. Irina S. Dolinskaya & Marina A. Epelman & Esra Şişikoğlu Sir & Robert L. Smith, 2016. "Parameter-Free Sampled Fictitious Play for Solving Deterministic Dynamic Programming Problems," Journal of Optimization Theory and Applications, Springer, vol. 169(2), pages 631-655, May.
    10. Rebecca S. Widrick & Sarah G. Nurre & Matthew J. Robbins, 2018. "Optimal Policies for the Management of an Electric Vehicle Battery Swap Station," Transportation Science, INFORMS, vol. 52(1), pages 59-79, January.
    11. Karl Inderfurth & Kampan Mukherjee, 2008. "Decision support for spare parts acquisition in post product life cycle," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 16(1), pages 17-42, March.
    12. Eugene A. Feinberg & Fenghsu Yang, 2016. "Optimal pricing for a GI/M/k/N queue with several customer types and holding costs," Queueing Systems: Theory and Applications, Springer, vol. 82(1), pages 103-120, February.
    13. Suresh Chand & Vernon Ning Hsu & Suresh Sethi, 2002. "Forecast, Solution, and Rolling Horizons in Operations Management Problems: A Classified Bibliography," Manufacturing & Service Operations Management, INFORMS, vol. 4(1), pages 25-43, September.
    14. M. Jacobson & N. Shimkin & A. Shwartz, 2003. "Markov Decision Processes with Slow Scale Periodic Decisions," Mathematics of Operations Research, INFORMS, vol. 28(4), pages 777-800, November.
    15. David Winkelmann & Matthias Ulrich & Michael Romer & Roland Langrock & Hermann Jahnke, 2022. "Dynamic Stochastic Inventory Management in E-Grocery Retailing," Papers 2205.06572, arXiv.org, revised Apr 2024.
    16. Iida, Tetsuo, 2001. "The infinite horizon non-stationary stochastic multi-echelon inventory problem and near-myopic policies," European Journal of Operational Research, Elsevier, vol. 134(3), pages 525-539, November.
    17. Ryan, Sarah M., 1998. "Forecast frequency in rolling horizon hedging heuristics for capacity expansion," European Journal of Operational Research, Elsevier, vol. 109(3), pages 550-558, September.
    18. Samorani, Michele & LaGanga, Linda R., 2015. "Outpatient appointment scheduling given individual day-dependent no-show predictions," European Journal of Operational Research, Elsevier, vol. 240(1), pages 245-257.
    19. Iida, Tetsuo, 1999. "The infinite horizon non-stationary stochastic inventory problem: Near myopic policies and weak ergodicity," European Journal of Operational Research, Elsevier, vol. 116(2), pages 405-422, July.
    20. Sinitskaya, Ekaterina, 2014. "Computational modeling of an economy using elements of artificial intelligence," ISU General Staff Papers 201401010800005291, Iowa State University, Department of Economics.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:199:y:2012:i:1:p:193-214:10.1007/s10479-012-1070-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.