IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v241y2024ics0951832023005422.html
   My bibliography  Save this article

Joint maintenance and spare part ordering from multiple suppliers for multicomponent systems using a deep reinforcement learning algorithm

Author

Listed:
  • Zheng, Meimei
  • Su, Zhiyun
  • Wang, Dong
  • Pan, Ershun

Abstract

This paper investigates the joint optimization of maintenance and spare part ordering from multiple suppliers for systems consisting of multiple components. When components degrade to poor conditions, they are replaced with spare parts if available. Spare parts can be purchased from multiple suppliers, each with distinct lead times, unit prices, and setup costs. To determine the optimal replacement and ordering decisions, this paper establishes a model through a Markov decision process and designs a value iteration algorithm to solve the model. However, the value iteration algorithm takes too much time to solve problems involving large numbers of components and suppliers. Thus, we design a hybrid deep reinforcement learning algorithm (HDRL) based on the reinforcement learning algorithm to solve large-scale problems. Numerical experiments are conducted to validate the effectiveness of the HDRL algorithm and analyze the joint decisions. The results show that, compared with the value iteration algorithm, the average cost gap is 3.86%, and the solving time can be reduced by at least 95.99% for systems with more than 2 suppliers and more than 3 components under the HDRL algorithm.

Suggested Citation

  • Zheng, Meimei & Su, Zhiyun & Wang, Dong & Pan, Ershun, 2024. "Joint maintenance and spare part ordering from multiple suppliers for multicomponent systems using a deep reinforcement learning algorithm," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    • Handle: RePEc:eee:reensy:v:241:y:2024:i:c:s0951832023005422
    DOI: 10.1016/j.ress.2023.109628
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832023005422
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2023.109628?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Jun & Zhu, Xiaoyan, 2021. "Joint optimization of condition-based maintenance and inventory control for a k-out-of-n:F system of multi-state degrading components," European Journal of Operational Research, Elsevier, vol. 290(2), pages 514-529.
    2. Wang, Xiaolin & Li, Lishuai & Xie, Min, 2020. "An unpunctual preventive maintenance policy under two-dimensional warranty," European Journal of Operational Research, Elsevier, vol. 282(1), pages 304-318.
    3. Zhang, Nan & Deng, Yingjun & Liu, Bin & Zhang, Jun, 2023. "Condition-based maintenance for a multi-component system in a dynamic operating environment," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    4. Shi, Yue & Zhu, Weihang & Xiang, Yisha & Feng, Qianmei, 2020. "Condition-based maintenance optimization for multi-component systems subject to a system reliability requirement," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    5. Yang, Ao & Qiu, Qingan & Zhu, Mingren & Cui, Lirong & Chen, Weilin & Chen, Jianhui, 2022. "Condition-based maintenance strategy for redundant systems with arbitrary structures using improved reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    6. Zhou, Xiaojun & Lu, Zhiqiang & Xi, Lifeng, 2012. "Preventive maintenance optimization for a multi-component system under changing job shop schedule," Reliability Engineering and System Safety, Elsevier, vol. 101(C), pages 14-20.
    7. Olde Keizer, Minou C.A. & Teunter, Ruud H. & Veldman, Jasper, 2017. "Joint condition-based maintenance and inventory optimization for systems with multiple components," European Journal of Operational Research, Elsevier, vol. 257(1), pages 209-222.
    8. Xiao Wang & Hongwei Wang & Chao Qi, 2016. "Multi-agent reinforcement learning based maintenance policy for a resource constrained flow line system," Journal of Intelligent Manufacturing, Springer, vol. 27(2), pages 325-333, April.
    9. Zhou, Yifan & Li, Bangcheng & Lin, Tian Ran, 2022. "Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    10. Liu, Xinbao & Yang, Tianji & Pei, Jun & Liao, Haitao & Pohl, Edward A., 2019. "Replacement and inventory control for a multi-customer product service system with decreasing replacement costs," European Journal of Operational Research, Elsevier, vol. 273(2), pages 561-574.
    11. Yan, Tao & Lei, Yaguo & Wang, Biao & Han, Tianyu & Si, Xiaosheng & Li, Naipeng, 2020. "Joint maintenance and spare parts inventory optimization for multi-unit systems considering imperfect maintenance actions," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    12. Caballé, N.C. & Castro, I.T. & Pérez, C.J. & Lanza-Gutiérrez, J.M., 2015. "A condition-based maintenance of a dependent degradation-threshold-shock model in a system with multiple degradation processes," Reliability Engineering and System Safety, Elsevier, vol. 134(C), pages 98-109.
    13. Zhang, Ning & Qi, Faqun & Zhang, Chengjie & Zhou, Hongming, 2022. "Joint optimization of condition-based maintenance policy and buffer capacity for a two-unit series system," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    14. Zhang, Jian-Xun & Du, Dang-Bo & Si, Xiao-Sheng & Hu, Chang-Hua & Zhang, Han-Wen, 2021. "Joint optimization of preventive maintenance and inventory management for standby systems with hybrid-deteriorating spare parts," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    15. Zhang, Nailong & Si, Wujun, 2020. "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    16. Qiu, Ruozhen & Sun, Minghe & Lim, Yun Fong, 2017. "Optimizing (s, S) policies for multi-period inventory models with demand distribution uncertainty: Robust dynamic programing approaches," European Journal of Operational Research, Elsevier, vol. 261(3), pages 880-892.
    17. Zhang, Xiaohong & Zeng, Jianchao, 2017. "Joint optimization of condition-based opportunistic maintenance and spare parts provisioning policy in multiunit systems," European Journal of Operational Research, Elsevier, vol. 262(2), pages 479-498.
    18. Zhao, Xian & He, Zongda & Wu, Yaguang & Qiu, Qingan, 2022. "Joint optimization of condition-based performance control and maintenance policies for mission-critical systems," Reliability Engineering and System Safety, Elsevier, vol. 226(C).
    19. Svoboda, Josef & Minner, Stefan & Yao, Man, 2021. "Typology and literature review on multiple supplier inventory control models," European Journal of Operational Research, Elsevier, vol. 293(1), pages 1-23.
    20. Oakley, Jordan L. & Wilson, Kevin J. & Philipson, Pete, 2022. "A condition-based maintenance policy for continuously monitored multi-component systems with economic and stochastic dependence," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    21. Karamatsoukis, C.C. & Kyriakidis, E.G., 2010. "Optimal maintenance of two stochastically deteriorating machines with an intermediate buffer," European Journal of Operational Research, Elsevier, vol. 207(1), pages 297-308, November.
    22. Sazvar, Z. & Mirzapour Al-e-hashem, S.M.J. & Baboli, A. & Akbari Jokar, M.R., 2014. "A bi-objective stochastic programming model for a centralized green supply chain with deteriorating products," International Journal of Production Economics, Elsevier, vol. 150(C), pages 140-154.
    23. Liang, Zhenglin & Liu, Bin & Xie, Min & Parlikad, Ajith Kumar, 2020. "Condition-based maintenance for long-life assets with exposure to operational and environmental risks," International Journal of Production Economics, Elsevier, vol. 221(C).
    24. Zheng, Meimei & Ye, Hongqing & Wang, Dong & Pan, Ershun, 2021. "Joint Optimization of Condition-Based Maintenance and Spare Parts Orders for Multi-Unit Systems with Dual Sourcing," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    25. Tian, Zhigang & Zhang, Han, 2022. "Wind farm predictive maintenance considering component level repairs and economic dependency," Renewable Energy, Elsevier, vol. 192(C), pages 495-506.
    26. Olde Keizer, Minou C.A. & Teunter, Ruud H. & Veldman, Jasper, 2016. "Clustering condition-based maintenance for systems with redundancy and economic dependencies," European Journal of Operational Research, Elsevier, vol. 251(2), pages 531-540.
    27. Xu, Jun & Liang, Zhenglin & Li, Yan-Fu & Wang, Kaibo, 2021. "Generalized condition-based maintenance optimization for multi-component systems considering stochastic dependency and imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    28. Andriotis, C.P. & Papakonstantinou, K.G., 2021. "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gan, Shuyuan & Hu, Hengheng & Coit, David W., 2023. "Maintenance optimization considering the mutual dependence of the environment and system with decreasing effects of imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    2. Zhu, Mixin & Zhou, Xiaojun, 2023. "Hybrid opportunistic maintenance policy for serial-parallel multi-station manufacturing systems with spare part overlap," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    3. Zhang, Wenyu & Zhang, Xiaohong & He, Shuguang & Zhao, Xing & He, Zhen, 2024. "Optimal condition-based maintenance policy for multi-component repairable systems with economic dependence in a finite-horizon," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    4. Xia, Tangbin & Cao, Lei & Xu, Yuhui & Zhang, Kaigan & Chen, Zhen & Pan, Ershun & Xi, Lifeng, 2024. "Multi-level maintenance and inventory joint optimization for a k-out-of-n hyper-system considering the selection of suppliers with incentive discount policies," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    5. de Jonge, Bram & Scarf, Philip A., 2020. "A review on maintenance optimization," European Journal of Operational Research, Elsevier, vol. 285(3), pages 805-824.
    6. Zhou, Yifan & Li, Bangcheng & Lin, Tian Ran, 2022. "Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    7. Zheng, Meimei & Ye, Hongqing & Wang, Dong & Pan, Ershun, 2021. "Joint Optimization of Condition-Based Maintenance and Spare Parts Orders for Multi-Unit Systems with Dual Sourcing," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    8. Azizi, Fariba & Salari, Nooshin, 2023. "A novel condition-based maintenance framework for parallel manufacturing systems based on bivariate birth/birth–death processes," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    9. Liu, Xinbao & Yang, Tianji & Pei, Jun & Liao, Haitao & Pohl, Edward A., 2019. "Replacement and inventory control for a multi-customer product service system with decreasing replacement costs," European Journal of Operational Research, Elsevier, vol. 273(2), pages 561-574.
    10. Mikhail, Mina & Ouali, Mohamed-Salah & Yacout, Soumaya, 2024. "A data-driven methodology with a nonparametric reliability method for optimal condition-based maintenance strategies," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    11. Lee, Dongkyu & Song, Junho, 2023. "Risk-informed operation and maintenance of complex lifeline systems using parallelized multi-agent deep Q-network," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    12. de Jonge, Bram, 2019. "Discretizing continuous-time continuous-state deterioration processes, with an application to condition-based maintenance optimization," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 1-5.
    13. Zheng, Meimei & Lin, Jie & Xia, Tangbin & Liu, Yu & Pan, Ershun, 2023. "Joint condition-based maintenance and spare provisioning policy for a K-out-of-N system with failures during inspection intervals," European Journal of Operational Research, Elsevier, vol. 308(3), pages 1220-1232.
    14. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    15. Zhu, Mixin & Zhou, Xiaojun, 2023. "Hierarchical-clustering-based joint optimization of spare part provision and maintenance scheduling for serial-parallel multi-station manufacturing systems," International Journal of Production Economics, Elsevier, vol. 264(C).
    16. Wang, Liying & Song, Yushuang & Zhang, Wenhua & Ling, Xiaoliang, 2023. "Condition-based inspection, component reallocation and replacement optimization of two-component interchangeable series system," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    17. Cha, Guesik & Park, Junseok & Moon, Ilkyeong, 2023. "Military aircraft flight and maintenance planning model considering heterogeneous maintenance tasks," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    18. Morato, P.G. & Andriotis, C.P. & Papakonstantinou, K.G. & Rigo, P., 2023. "Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    19. Wei, Shuaichong & Nourelfath, Mustapha & Nahas, Nabil, 2023. "Analysis of a production line subject to degradation and preventive maintenance," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    20. Zhu, Mixin & Zhou, Xiaojun, 2022. "Hypergraph-based joint optimization of spare part provision and maintenance scheduling for serial-parallel multi-station manufacturing systems," Reliability Engineering and System Safety, Elsevier, vol. 225(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:241:y:2024:i:c:s0951832023005422. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.