IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v241y2024ics0951832023004969.html
   My bibliography  Save this article

Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach

Author

Listed:
  • Tseremoglou, Iordanis
  • Santos, Bruno F.

Abstract

In the Condition-Based Maintenance (CBM) context, the definition of optimal maintenance plans for an aircraft fleet depends on an efficient integration of : (i) the probabilistic predictions of the health condition of the components and (ii) the stochastic arrival of the corrective maintenance tasks, together with consideration of the preventive maintenance tasks as defined in the Maintenance Planning Document (MPD) . To this end, in this paper, we present a two-stage dynamic scheduling framework to solve the aircraft fleet maintenance scheduling problem under a CBM strategy in a disruptive environment. In the first stage of the framework, we address the uncertainty in the predicted health state of the monitored components by planning the optimal maintenance policy based upon the belief state-space of the health of the components. The decision-making process is formulated as a Partially Observable Markov Decision Process (POMDP) and is solved using the Partially Observable Monte Carlo Planning (POMCP) algorithm, considering the aircraft maintenance scheduling problem requirements. In the second stage, a Deep Q-Network (DQN) is developed, that integrates the defined maintenance policy of the monitored components within the scheduling of the aircraft fleet’s preventive and corrective maintenance tasks. Our model, through a rolling horizon approach, continuously creates and adjusts the maintenance schedule, reacting to new updated task information, where the availability of maintenance resources constraints the execution of each task. The proposed framework was tested on a case study from a large airline and the performance was evaluated against the current state practice of the airline. The results show that our model can schedule 96.4% of monitored components on-time. As a consequence of this, a 46.2% maintenance cost reduction is achieved for the considered monitored components relative to a corrective maintenance approach.

Suggested Citation

  • Tseremoglou, Iordanis & Santos, Bruno F., 2024. "Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    • Handle: RePEc:eee:reensy:v:241:y:2024:i:c:s0951832023004969
    DOI: 10.1016/j.ress.2023.109582
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832023004969
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2023.109582?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Andriotis, C.P. & Papakonstantinou, K.G., 2019. "Managing engineering systems with large state and action spaces through deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    2. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    3. Nguyen, Khanh T. P. & Do, Phuc & Huynh, Khac Tuan & Bérenguer, Christophe & Grall, Antoine, 2019. "Joint optimization of monitoring quality and replacement decisions in condition-based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 189(C), pages 177-195.
    4. Papakonstantinou, K.G. & Shinozuka, M., 2014. "Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementation," Reliability Engineering and System Safety, Elsevier, vol. 130(C), pages 214-224.
    5. Zhao, Yunfei & Smidts, Carol, 2022. "Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    6. Liu, Xingchen & Sun, Qiuzhuang & Ye, Zhi-Sheng & Yildirim, Murat, 2021. "Optimal multi-type inspection policy for systems with imperfect online monitoring," Reliability Engineering and System Safety, Elsevier, vol. 207(C).
    7. Alaswad, Suzan & Xiang, Yisha, 2017. "A review on condition-based maintenance optimization models for stochastically deteriorating system," Reliability Engineering and System Safety, Elsevier, vol. 157(C), pages 54-63.
    8. Deng, Qichen & Santos, Bruno F. & Curran, Richard, 2020. "A practical dynamic programming based methodology for aircraft maintenance check scheduling optimization," European Journal of Operational Research, Elsevier, vol. 281(2), pages 256-273.
    9. Zhang, Nailong & Si, Wujun, 2020. "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    10. J.P. Sprong & X. Jiang & H. Polinder, 2020. "Deployment of Prognostics to Optimize Aircraft Maintenance – A Literature Review," Journal of International Business Research and Marketing, Inovatus Services Ltd., vol. 5(4), pages 26-37, May.
    11. Meissner, Robert & Rahn, Antonia & Wicke, Kai, 2021. "Developing prescriptive maintenance strategies in the aviation industry based on a discrete-event simulation framework for post-prognostics decision making," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    12. Carlos Lagos & Felipe Delgado & Mathias A. Klapp, 2020. "Dynamic Optimization for Airline Maintenance Operations," Transportation Science, INFORMS, vol. 54(4), pages 998-1015, July.
    13. Zhao, Xufeng & Al-Khalifa, Khalifa N. & Magid Hamouda, Abdel & Nakagawa, Toshio, 2017. "Age replacement models: A summary with new perspectives and methods," Reliability Engineering and System Safety, Elsevier, vol. 161(C), pages 95-105.
    14. Song, Chaolin & Zhang, Chi & Shafieezadeh, Abdollah & Xiao, Rucheng, 2022. "Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approach," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    15. de Pater, Ingeborg & Mitici, Mihaela, 2021. "Predictive maintenance for multi-component systems of repairables with Remaining-Useful-Life prognostics and a limited stock of spare components," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    16. Andriotis, C.P. & Papakonstantinou, K.G., 2021. "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Najafi, Seyedvahid & Lee, Chi-Guhn, 2023. "A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    2. Lee, Dongkyu & Song, Junho, 2023. "Risk-informed operation and maintenance of complex lifeline systems using parallelized multi-agent deep Q-network," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    3. Morato, P.G. & Andriotis, C.P. & Papakonstantinou, K.G. & Rigo, P., 2023. "Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    4. Lee, Juseong & Mitici, Mihaela, 2023. "Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    5. Zhang, Qin & Liu, Yu & Xiahou, Tangfan & Huang, Hong-Zhong, 2023. "A heuristic maintenance scheduling framework for a military aircraft fleet under limited maintenance capacities," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    6. KarabaÄŸ, Oktay & Eruguz, Ayse Sena & Basten, Rob, 2020. "Integrated optimization of maintenance interventions and spare part selection for a partially observable multi-component system," Reliability Engineering and System Safety, Elsevier, vol. 200(C).
    7. Mikhail, Mina & Ouali, Mohamed-Salah & Yacout, Soumaya, 2024. "A data-driven methodology with a nonparametric reliability method for optimal condition-based maintenance strategies," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    8. Zhao, Yunfei & Smidts, Carol, 2022. "Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    9. Arcieri, Giacomo & Hoelzl, Cyprien & Schwery, Oliver & Straub, Daniel & Papakonstantinou, Konstantinos G. & Chatzi, Eleni, 2023. "Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    10. Nguyen, Van-Thai & Do, Phuc & Vosin, Alexandre & Iung, Benoit, 2022. "Artificial-intelligence-based maintenance decision-making and optimization for multi-state component systems," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    11. Xu, Zhaoyi & Saleh, Joseph Homer, 2021. "Machine learning for reliability engineering and safety applications: Review of current status and future opportunities," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    12. Kıvanç, İpek & Özgür-Ünlüakın, Demet & Bilgiç, Taner, 2022. "Maintenance policy analysis of the regenerative air heater system using factored POMDPs," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    13. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    14. Cha, Guesik & Park, Junseok & Moon, Ilkyeong, 2023. "Military aircraft flight and maintenance planning model considering heterogeneous maintenance tasks," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    15. Guo, Chunhui & Liang, Zhenglin, 2022. "A predictive Markov decision process for optimizing inspection and maintenance strategies of partially observable multi-state systems," Reliability Engineering and System Safety, Elsevier, vol. 226(C).
    16. Ye, Zhenggeng & Cai, Zhiqiang & Yang, Hui & Si, Shubin & Zhou, Fuli, 2023. "Joint optimization of maintenance and quality inspection for manufacturing networks based on deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    17. Pedersen, Tom Ivar & Vatn, Jørn, 2022. "Optimizing a condition-based maintenance policy by taking the preferences of a risk-averse decision maker into account," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    18. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    19. Andersen, Jesper Fink & Andersen, Anders Reenberg & Kulahci, Murat & Nielsen, Bo Friis, 2022. "A numerical study of Markov decision process algorithms for multi-component replacement problems," European Journal of Operational Research, Elsevier, vol. 299(3), pages 898-909.
    20. Mitici, Mihaela & de Pater, Ingeborg & Barros, Anne & Zeng, Zhiguo, 2023. "Dynamic predictive maintenance for multiple components using data-driven probabilistic RUL prognostics: The case of turbofan engines," Reliability Engineering and System Safety, Elsevier, vol. 234(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:241:y:2024:i:c:s0951832023004969. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.