IDEAS home Printed from https://ideas.repec.org/a/spr/jcomop/v49y2025i3d10.1007_s10878-025-01272-9.html
   My bibliography  Save this article

Superposed semi-Markov decision process with application to optimal maintenance systems

Author

Listed:
  • Jianmin Shi

    (Wuhan University
    Haitong Securities)

Abstract

This paper investigates the superposition problem of two or more individual semi-Markov decision processes (SMDPs). The new sequential decision process superposed by individual SMDPs is no longer an SMDP and cannot be handled by routine iterative algorithms, but we can expand its state spaces to obtain a hybrid-state SMDP. Using this hybrid-state SMDP as an auxiliary and inspired by the Robbins–Monro algorithm underlying the reinforcement learning method, we propose an iteration algorithm based on a combination of dynamic programming and reinforcement learning to numerically solve the superposed sequential decision problem. As an illustration example, we apply our superposition model and algorithm to solve the optimal maintenance problem of a two-component independent parallel system.

Suggested Citation

  • Jianmin Shi, 2025. "Superposed semi-Markov decision process with application to optimal maintenance systems," Journal of Combinatorial Optimization, Springer, vol. 49(3), pages 1-19, April.
    • Handle: RePEc:spr:jcomop:v:49:y:2025:i:3:d:10.1007_s10878-025-01272-9
    DOI: 10.1007/s10878-025-01272-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10878-025-01272-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10878-025-01272-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Richard F. Serfozo, 1979. "Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes," Operations Research, INFORMS, vol. 27(3), pages 616-620, June.
    2. Daniel McInnes & Boris Miller & Gregory Miller & Sergei Schreider, 2020. "Towards Tensor Representation of Controlled Coupled Markov Chains," Mathematics, MDPI, vol. 8(10), pages 1-17, October.
    3. Jack P. C. Kleijnen, 2015. "Simulation Optimization," International Series in Operations Research & Management Science, in: Design and Analysis of Simulation Experiments, edition 2, chapter 6, pages 241-300, Springer.
    4. Dimitris Bertsimas & Velibor V. Mišić, 2016. "Decomposable Markov Decision Processes: A Fluid Optimization Approach," Operations Research, INFORMS, vol. 64(6), pages 1537-1555, December.
    5. Huang, Yonghui & Guo, Xianping, 2011. "Finite horizon semi-Markov decision processes with application to maintenance systems," European Journal of Operational Research, Elsevier, vol. 212(1), pages 131-140, July.
    6. Irene Votsi & Nikolaos Limnios & George Tsaklidis & Eleftheria Papadimitriou, 2012. "Estimation of the Expected Number of Earthquake Occurrences Based on Semi-Markov Models," Methodology and Computing in Applied Probability, Springer, vol. 14(3), pages 685-703, September.
    7. Milan Kumar Das & Anindya Goswami & Nimit Rana, 2016. "Risk Sensitive Portfolio Optimization in a Jump Diffusion Model with Regimes," Papers 1603.09149, arXiv.org, revised Jan 2018.
    8. David B. Brown & Jingwei Zhang, 2022. "Dynamic Programs with Shared Resources and Signals: Dynamic Fluid Policies and Asymptotic Optimality," Operations Research, INFORMS, vol. 70(5), pages 3015-3033, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Milan Kumar Das & Anindya Goswami, 2019. "Testing of binary regime switching models using squeeze duration analysis," International Journal of Financial Engineering (IJFE), World Scientific Publishing Co. Pte. Ltd., vol. 6(01), pages 1-20, March.
    2. Votsi, I. & Limnios, N. & Tsaklidis, G. & Papadimitriou, E., 2013. "Hidden Markov models revealing the stress field underlying the earthquake generation," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 392(13), pages 2868-2885.
    3. Deligiannis, Michalis & Liberopoulos, George, 2023. "Dynamic ordering and buyer selection policies when service affects future demand," Omega, Elsevier, vol. 118(C).
    4. Xiuli Chao & Frank Y. Chen, 2005. "An Optimal Production and Shutdown Strategy when a Supplier Offers an Incentive Program," Manufacturing & Service Operations Management, INFORMS, vol. 7(2), pages 130-143, March.
    5. Dwi Ertiningsih & Sandjai Bhulai & Flora Spieksma, 2018. "A novel use of value iteration for deriving bounds for threshold and switching curve optimal policies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 65(8), pages 638-659, December.
    6. Nicole Leder & Bernd Heidergott & Arie Hordijk, 2010. "An Approximation Approach for the Deviation Matrix of Continuous-Time Markov Processes with Application to Markov Decision Theory," Operations Research, INFORMS, vol. 58(4-part-1), pages 918-932, August.
    7. Laura Eslava & Fernando Baltazar-Larios & Bor Reynoso, 2022. "Maximum Likelihood Estimation for a Markov-Modulated Jump-Diffusion Model," Papers 2211.17220, arXiv.org.
    8. Milan Kumar Das & Anindya Goswami & Sharan Rajani, 2019. "Inference of Binary Regime Models with Jump Discontinuities," Papers 1910.10606, arXiv.org, revised Mar 2022.
    9. Lijun Bo & Huafu Liao & Xiang Yu, 2017. "Risk Sensitive Portfolio Optimization with Default Contagion and Regime-Switching," Papers 1712.05676, arXiv.org, revised Oct 2018.
    10. Md. Asaduzzaman & A. Latif, 2014. "A parametric Markov renewal model for predicting tropical cyclones in Bangladesh," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 73(2), pages 597-612, September.
    11. Milan Kumar Das & Anindya Goswami, 2018. "Testing of Binary Regime Switching Models using Squeeze Duration Analysis," Papers 1807.04393, arXiv.org, revised Aug 2018.
    12. Danisman, Ozgur & Uzunoglu Kocer, Umay, 2021. "Hidden Markov models with binary dependence," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    13. Dinard van der Laan, 2015. "Assigning Multiple Job Types to Parallel Specialized Servers," Tinbergen Institute Discussion Papers 15-102/III, Tinbergen Institute.
    14. Insua, David Rios & Ruggeri, Fabrizio & Soyer, Refik & Wilson, Simon, 2020. "Advances in Bayesian decision making in reliability," European Journal of Operational Research, Elsevier, vol. 282(1), pages 1-18.
    15. Sergei V. Savin & Morris A. Cohen & Noah Gans & Ziv Katalan, 2005. "Capacity Management in Rental Businesses with Two Customer Bases," Operations Research, INFORMS, vol. 53(4), pages 617-631, August.
    16. D’Amico, Guglielmo & Petroni, Filippo, 2023. "ROCOF of higher order for semi-Markov processes," Applied Mathematics and Computation, Elsevier, vol. 441(C).
    17. Sadoghi, Amirhossein & Vecer, Jan, 2022. "Optimal liquidation problem in illiquid markets," European Journal of Operational Research, Elsevier, vol. 296(3), pages 1050-1066.
    18. Xufeng Yang & Juliang Zhang & Wen Jiao & Hong Yan, 2023. "Optimal Capacity Rationing Policy for a Container Leasing System with Multiple Kinds of Customers and Substitutable Containers," Management Science, INFORMS, vol. 69(3), pages 1468-1485, March.
    19. Behice Meltem Kayhan & Gokalp Yildiz, 2023. "Reinforcement learning applications to machine scheduling problems: a comprehensive literature review," Journal of Intelligent Manufacturing, Springer, vol. 34(3), pages 905-929, March.
    20. Vlad Stefan Barbu & Nicolas Vergne, 2019. "Reliability and Survival Analysis for Drifting Markov Models: Modeling and Estimation," Methodology and Computing in Applied Probability, Springer, vol. 21(4), pages 1407-1429, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcomop:v:49:y:2025:i:3:d:10.1007_s10878-025-01272-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.