IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v244y2024ics0951832023008359.html

Maintenance optimization of multi-unit balanced systems using deep reinforcement learning

Author

Listed:
  • Liu, Hengchang
  • Li, Bo
  • Yao, Fengming
  • Hu, Gexi
  • Xie, Lei

Abstract

Balanced systems are extensively utilized in various engineering fields. However, the study of maintenance optimization for balanced systems under dependent competing risks is very limited, and the existing method for orthodox multi-unit systems cannot solve the maintenance problem of balanced systems. Thus, we present a novel maintenance optimization method for systems with multiple balanced units. The units in the system are assumed to experience multiple failure processes due to continuous degradation and external shock damage. The system failure occurs in the following situations: (1) the deterioration levels of units on the symmetric positions exceed critical thresholds. (2) the deterioration level of any unit reaches the failure threshold. (3) the shock magnitude is greater than the hard failure threshold. By employing a customized deep reinforcement learning algorithm, we derive the optimal maintenance policy under a fixed planning horizon. Notably, our approach differs from previous research on maintenance optimization for balanced systems, the proposed method avoids the discretizing of the deterioration state. This significantly enhances computational efficiency, particularly when applied to multi-unit balanced systems. Numerous numerical studies illustrate the effectiveness of our proposed maintenance optimization method for multi-unit balanced systems.

Suggested Citation

  • Liu, Hengchang & Li, Bo & Yao, Fengming & Hu, Gexi & Xie, Lei, 2024. "Maintenance optimization of multi-unit balanced systems using deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    • Handle: RePEc:eee:reensy:v:244:y:2024:i:c:s0951832023008359
    DOI: 10.1016/j.ress.2023.109921
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832023008359
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2023.109921?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    2. Wang, Xiaoyue & Ning, Ru & Zhao, Xian & Wu, Congshan, 2023. "Reliability assessments for two types of balanced systems with multi-state protective devices," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    3. Yangyao, Shi & Xinchen, Zhuang & Tianxiang, Yu & Zijian, Zhang, 2023. "Multi-state balance system reliability research considering load influence," Reliability Engineering and System Safety, Elsevier, vol. 233(C).
    4. Wang, Jingjing & Miao, Yonghao, 2021. "Optimal preventive maintenance policy of the balanced system under the semi-Markov model," Reliability Engineering and System Safety, Elsevier, vol. 213(C).
    5. Dingguo Hua & Elsayed A. Elsayed, 2018. "Reliability approximation of k-out-of-n pairs: G balanced systems with spatially distributed units," IISE Transactions, Taylor & Francis Journals, vol. 50(7), pages 616-626, July.
    6. Najafi, Seyedvahid & Zheng, Rui & Lee, Chi-Guhn, 2021. "An optimal opportunistic maintenance policy for a two-unit series system with general repair using proportional hazards models," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    7. Yang, Ao & Qiu, Qingan & Zhu, Mingren & Cui, Lirong & Chen, Weilin & Chen, Jianhui, 2022. "Condition-based maintenance strategy for redundant systems with arbitrary structures using improved reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    8. Olde Keizer, Minou C.A. & Flapper, Simme Douwe P. & Teunter, Ruud H., 2017. "Condition-based maintenance policies for systems with multiple dependent components: A review," European Journal of Operational Research, Elsevier, vol. 261(2), pages 405-420.
    9. Wang, Jingjing & Zheng, Rui & Lin, Tianran, 2022. "Maintenance modeling for balanced systems subject to two competing failure modes," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    10. Fang, Chen & Cui, Lirong, 2020. "Reliability analysis for balanced engine systems with m sectors by considering start-up probability," Reliability Engineering and System Safety, Elsevier, vol. 197(C).
    11. Zhang, Nailong & Si, Wujun, 2020. "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    12. Zhao, Xian & Wang, Siqi & Wang, Xiaoyue & Fan, Yu, 2020. "Multi-state balanced systems in a shock environment," Reliability Engineering and System Safety, Elsevier, vol. 193(C).
    13. Zheng, Meimei & Ye, Hongqing & Wang, Dong & Pan, Ershun, 2021. "Joint Optimization of Condition-Based Maintenance and Spare Parts Orders for Multi-Unit Systems with Dual Sourcing," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    14. Wang, Jingjing & Wang, Zongxi & Zheng, Rui, 2021. "Optimal inventory policy for a balanced system subject to hard failure," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    15. Hu, Jiawen & Shen, Jingyuan & Shen, Lijuan, 2020. "Opportunistic maintenance for two-component series systems subject to dependent degradation and shock," Reliability Engineering and System Safety, Elsevier, vol. 201(C).
    16. Wang, Jingjing & Qiu, Qingan & Wang, Huanhuan & Lin, Cong, 2021. "Optimal condition-based preventive maintenance policy for balanced systems," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    17. Lirong Cui & Jianhui Chen & Xiangchen Li, 2019. "Balanced reliability systems under Markov processes," IISE Transactions, Taylor & Francis Journals, vol. 51(9), pages 1025-1035, September.
    18. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    19. van Noortwijk, J.M., 2009. "A survey of the application of gamma processes in maintenance," Reliability Engineering and System Safety, Elsevier, vol. 94(1), pages 2-21.
    20. Wang, Siqi & Zhao, Xian & Wu, Congshan & Wang, Xiaoyue, 2023. "Joint optimization of multi-stage component reassignment and preventive maintenance for balanced systems considering imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    21. Wang, Xiaoyue & Zhao, Xian & Wang, Siqi & Sun, Leping, 2020. "Reliability and maintenance for performance-balanced systems operating in a shock environment," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zhang, Wenyu & He, Shuguang & Zhang, Xiaohong & Zhao, Xing, 2024. "Joint optimization of job scheduling, condition-based maintenance planning, and spare parts ordering for degrading production systems," Reliability Engineering and System Safety, Elsevier, vol. 252(C).
    2. Zheng, Rui & Fang, Haojun & Song, Yanying, 2024. "A condition-based maintenance policy for a two-component balanced system with dependent degradation processes," Reliability Engineering and System Safety, Elsevier, vol. 252(C).
    3. Fang, Haojun & Zheng, Rui & Xia, Xiangde & Hu, Chiwei, 2025. "Condition-based maintenance policy for a two-component balanced system with a multimode protective device," Reliability Engineering and System Safety, Elsevier, vol. 262(C).
    4. Wang, Siqi & Li, Songqi & Zhao, Xian & Wu, Congshan, 2025. "Joint optimization of multi-stage work intensity selecting and maintenance policy for a two-dimensional balanced system," Reliability Engineering and System Safety, Elsevier, vol. 264(PA).
    5. Andersen, Jesper Fink & Nielsen, Bo Friis, 2025. "A comparative study of time-based maintenance and condition-based maintenance for multi-component systems," Reliability Engineering and System Safety, Elsevier, vol. 256(C).
    6. Zhao, Sangqi & Wei, Yian & Li, Yang & Cheng, Yao, 2026. "A multi-agent reinforcement learning (MARL) framework for designing an optimal state-specific hybrid maintenance policy for a series k-out-of-n load-sharing system," Reliability Engineering and System Safety, Elsevier, vol. 265(PA).
    7. Geurtsen, M. & Leenen, C. & Adan, I. & Atan, Z., 2026. "Deep reinforcement learning for optimal planning of production line maintenance with deterioration," Reliability Engineering and System Safety, Elsevier, vol. 266(PB).
    8. Sun, Yudao & Yin, Juan, 2025. "Dynamic condition-based maintenance for shock systems based on damage evolutions using deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 261(C).
    9. Wang, Jingjing & Luo, Lingyun & Jin, Yuxue & Yang, Li, 2025. "Joint optimization of maintenance policy and two-way stock transshipments policy for balanced systems," Reliability Engineering and System Safety, Elsevier, vol. 264(PA).
    10. Zhao, Sangqi & Wei, Yian & Cheng, Yao & Li, Yang, 2025. "A state-specific joint size, maintenance, and inventory policy for a k-out-of-n load-sharing system subject to self-announcing failures," Reliability Engineering and System Safety, Elsevier, vol. 257(PB).
    11. Xu, Dong & Tian, Yubin & Wang, Dianpeng & Shi, Junbiao, 2025. "Reliability analysis for a dynamic k-out-of-n system with thermal balance control," Reliability Engineering and System Safety, Elsevier, vol. 262(C).
    12. Zhu, Yunxin & Zheng, Meimei & Su, Zhiyun & Xia, Tangbin & Lin, Jie & Pan, Ershun, 2025. "Deep reinforcement learning for joint optimization of maintenance and spare parts ordering considering spare parts supply uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 264(PB).
    13. Cheraghi, Morteza & Taghipour, Sharareh, 2025. "Safety Integrity Level (SIL) evaluation of safety instrumented systems considering competing failure modes and subsystem priorities," Reliability Engineering and System Safety, Elsevier, vol. 260(C).
    14. Chen, Jiangxi & Zhou, Xiaojun, 2026. "A GNN-boosted reinforcement learning framework for maintenance optimization in multi-dependency manufacturing system," Reliability Engineering and System Safety, Elsevier, vol. 269(C).
    15. Chen, Jiangxi & Zhou, Xiaojun, 2025. "Reinforcement learning based maintenance scheduling of flexible multi-machine manufacturing systems with varying interactive degradation," Reliability Engineering and System Safety, Elsevier, vol. 260(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zheng, Rui & Fang, Haojun & Song, Yanying, 2024. "A condition-based maintenance policy for a two-component balanced system with dependent degradation processes," Reliability Engineering and System Safety, Elsevier, vol. 252(C).
    2. Chai, Xiaofei & Kilic, Onur A. & Veldman, Jasper & Teunter, Ruud H. & Zhao, Xian, 2024. "Condition-based reallocation and maintenance for a 1-out-of-2 pairs balanced system," European Journal of Operational Research, Elsevier, vol. 318(2), pages 618-628.
    3. Dong, Bingchen & Liang, Zhenglin, 2026. "A novel method for reliability design of multi-tier circular (k,Smin)-out-of-(n1,…,nm,S): G balanced systems subject to external disturbances," Reliability Engineering and System Safety, Elsevier, vol. 266(PA).
    4. Wang, Jingjing & Luo, Lingyun & Jin, Yuxue & Yang, Li, 2025. "Joint optimization of maintenance policy and two-way stock transshipments policy for balanced systems," Reliability Engineering and System Safety, Elsevier, vol. 264(PA).
    5. Xu, Dong & Tian, Yubin & Wang, Dianpeng & Shi, Junbiao, 2025. "Reliability analysis for a dynamic k-out-of-n system with thermal balance control," Reliability Engineering and System Safety, Elsevier, vol. 262(C).
    6. Baik, Seung Min & Cho, Yongkyu, 2026. "Lifetime analysis of circular k-out-of-n: G balanced systems in a shock environment," Reliability Engineering and System Safety, Elsevier, vol. 265(PB).
    7. Fang, Haojun & Zheng, Rui & Xia, Xiangde & Hu, Chiwei, 2025. "Condition-based maintenance policy for a two-component balanced system with a multimode protective device," Reliability Engineering and System Safety, Elsevier, vol. 262(C).
    8. Wu, Congshan & Zhao, Xian & Wang, Siqi & Song, Yanbo, 2022. "Reliability analysis of consecutive-k-out-of-r-from-n subsystems: F balanced systems with load sharing," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    9. Gao, Yingna & Wu, Bei & Li, Meiyan & Fang, Chen, 2026. "Reliability modeling and analysis of multi-state balanced systems with dual balance restriction mechanism," Reliability Engineering and System Safety, Elsevier, vol. 269(C).
    10. Gan, Shuyuan & Hu, Hengheng & Coit, David W., 2023. "Maintenance optimization considering the mutual dependence of the environment and system with decreasing effects of imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    11. Wang, Xiaoyue & Ning, Ru & Zhao, Xian & Wu, Congshan, 2023. "Reliability assessments for two types of balanced systems with multi-state protective devices," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    12. Zhao, Xian & Wang, Chen & Wang, Siqi & Han, He, 2025. "Standby component replacement strategy for a balanced system with a standby pool," Reliability Engineering and System Safety, Elsevier, vol. 254(PB).
    13. Wang, Siqi & Li, Shuyun & Mo, Lipo, 2025. "Joint optimization of condition-based maintenance and component reallocation for phased-mission balanced systems with flexible structure considering imperfect substitution," Reliability Engineering and System Safety, Elsevier, vol. 264(PB).
    14. Zhao, Xian & Han, He & Jiao, Chunhui & Qiu, Qingan, 2024. "Reliability modeling of k-out-of-n: F balanced systems with common bus performance sharing," Reliability Engineering and System Safety, Elsevier, vol. 248(C).
    15. Wang, Jingjing & Zheng, Rui & Lin, Tianran, 2022. "Maintenance modeling for balanced systems subject to two competing failure modes," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    16. Yangyao, Shi & Xinchen, Zhuang & Tianxiang, Yu & Zijian, Zhang, 2023. "Multi-state balance system reliability research considering load influence," Reliability Engineering and System Safety, Elsevier, vol. 233(C).
    17. Wang, Xiaoyue & Ning, Ru & Zhao, Xian & Zhou, Jian, 2022. "Reliability analyses of k-out-of-n: F capability-balanced systems in a multi-source shock environment," Reliability Engineering and System Safety, Elsevier, vol. 227(C).
    18. Wang, Siqi & Zhao, Xian & Wu, Congshan & Wang, Xiaoyue, 2023. "Joint optimization of multi-stage component reassignment and preventive maintenance for balanced systems considering imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    19. Wang, Jingjing & Qiu, Qingan & Wang, Huanhuan & Lin, Cong, 2021. "Optimal condition-based preventive maintenance policy for balanced systems," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    20. Zhou, Yifan & Li, Bangcheng & Lin, Tian Ran, 2022. "Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 217(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:244:y:2024:i:c:s0951832023008359. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.