IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v235y2023ics0951832023001461.html
   My bibliography  Save this article

A sequential decision problem formulation and deep reinforcement learning solution of the optimization of O&M of cyber-physical energy systems (CPESs) for reliable and safe power production and supply

Author

Listed:
  • Hao, Zhaojun
  • Di Maio, Francesco
  • Zio, Enrico

Abstract

The Operation & Maintenance (O&M) of Cyber-Physical Energy Systems (CPESs) is driven by reliable and safe production and supply, that need to account for flexibility to respond to the uncertainty in energy demand and also supply due to the stochasticity of Renewable Energy Sources (RESs); at the same time, accidents of severe consequences must be avoided for safety reasons. In this paper, we consider O&M strategies for CPES reliable and safe production and supply, and develop a Deep Reinforcement Learning (DRL) approach to search for the best strategy, considering the system components health conditions, their Remaining Useful Life (RUL), and possible accident scenarios. The approach integrates Proximal Policy Optimization (PPO) and Imitation Learning (IL) for training RL agent, with a CPES model that embeds the components RUL estimator and their failure process model. The novelty of the work lies in i) taking production plan into O&M decisions to implement maintenance and operate flexibly; ii) embedding the reliability model into CPES model to recognize safety related components and set proper maintenance RUL thresholds. An application, the Advanced Lead-cooled Fast Reactor European Demonstrator (ALFRED), is provided. The optimal solution found by DRL is shown to outperform those provided by state-of-the-art O&M policies.

Suggested Citation

  • Hao, Zhaojun & Di Maio, Francesco & Zio, Enrico, 2023. "A sequential decision problem formulation and deep reinforcement learning solution of the optimization of O&M of cyber-physical energy systems (CPESs) for reliable and safe power production and supply," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    • Handle: RePEc:eee:reensy:v:235:y:2023:i:c:s0951832023001461
    DOI: 10.1016/j.ress.2023.109231
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832023001461
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2023.109231?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Baraldi, Piero & Mangili, Francesca & Zio, Enrico, 2013. "Investigation of uncertainty treatment capability of model-based and data-driven prognostic methods using simulated data," Reliability Engineering and System Safety, Elsevier, vol. 112(C), pages 94-108.
    2. Zhaojun Hao & Francesco Di Maio & Enrico Zio, 2021. "Multi-State Reliability Assessment Model of Base-Load Cyber-Physical Energy Systems (CPES) during Flexible Operation Considering the Aging of Cyber Components," Energies, MDPI, vol. 14(11), pages 1-18, June.
    3. Xi-liang Chen & Lei Cao & Chen-xi Li & Zhi-xiong Xu & Jun Lai, 2018. "Ensemble Network Architecture for Deep Reinforcement Learning," Mathematical Problems in Engineering, Hindawi, vol. 2018, pages 1-6, April.
    4. Zhang, Sai & Du, Mengyu & Tong, Jiejuan & Li, Yan-Fu, 2019. "Multi-objective optimization of maintenance program in multi-unit nuclear power plant sites," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 532-548.
    5. Pinciroli, Luca & Baraldi, Piero & Ballabio, Guido & Compare, Michele & Zio, Enrico, 2022. "Optimization of the Operation and Maintenance of renewable energy systems by Deep Reinforcement Learning," Renewable Energy, Elsevier, vol. 183(C), pages 752-763.
    6. Zhang, Jiusi & Jiang, Yuchen & Li, Xiang & Huo, Mingyi & Luo, Hao & Yin, Shen, 2022. "An adaptive remaining useful life prediction approach for single battery with unlabeled small sample data and parameter uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    7. Wang, Wei & Di Maio, Francesco & Zio, Enrico, 2020. "Considering the human operator cognitive process for the interpretation of diagnostic outcomes related to component failures and cyber security attacks," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    8. Zhipeng Liang & Hao Chen & Junhao Zhu & Kangkang Jiang & Yanran Li, 2018. "Adversarial Deep Reinforcement Learning in Portfolio Management," Papers 1808.09940, arXiv.org, revised Nov 2018.
    9. Saleh, Ali & Chiachío, Manuel & Salas, Juan Fernández & Kolios, Athanasios, 2023. "Self-adaptive optimized maintenance of offshore wind turbines by intelligent Petri nets," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    10. Wang, Wei & Cammi, Antonio & Di Maio, Francesco & Lorenzi, Stefano & Zio, Enrico, 2018. "A Monte Carlo-based exploration framework for identifying components vulnerable to cyber threats in nuclear power plants," Reliability Engineering and System Safety, Elsevier, vol. 175(C), pages 24-37.
    11. Nguyen, Van-Thai & Do, Phuc & Vosin, Alexandre & Iung, Benoit, 2022. "Artificial-intelligence-based maintenance decision-making and optimization for multi-state component systems," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    12. Zio, Enrico & Di Maio, Francesco, 2010. "A data-driven fuzzy approach for predicting the remaining useful life in dynamic failure scenarios of a nuclear system," Reliability Engineering and System Safety, Elsevier, vol. 95(1), pages 49-57.
    13. Zhang, Jiusi & Li, Xiang & Tian, Jilun & Jiang, Yuchen & Luo, Hao & Yin, Shen, 2023. "A variational local weighted deep sub-domain adaptation network for remaining useful life prediction facing cross-domain condition," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    14. Zio, Enrico, 2016. "Challenges in the vulnerability and risk analysis of critical infrastructures," Reliability Engineering and System Safety, Elsevier, vol. 152(C), pages 137-150.
    15. Fan, Lin & Su, Huai & Wang, Wei & Zio, Enrico & Zhang, Li & Yang, Zhaoming & Peng, Shiliang & Yu, Weichao & Zuo, Lili & Zhang, Jinjun, 2022. "A systematic method for the optimization of gas supply reliability in natural gas pipeline network based on Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    16. Ganesh, Akhil Hannegudda & Xu, Bin, 2022. "A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution," Renewable and Sustainable Energy Reviews, Elsevier, vol. 154(C).
    17. Zhang, Jiusi & Jiang, Yuchen & Wu, Shimeng & Li, Xiang & Luo, Hao & Yin, Shen, 2022. "Prediction of remaining useful life based on bidirectional gated recurrent unit with temporal self-attention mechanism," Reliability Engineering and System Safety, Elsevier, vol. 221(C).
    18. Zio, Enrico, 2022. "Prognostics and Health Management (PHM): Where are we and where do we (need to) go in theory and practice," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    19. Pierobon, Leonardo & Casati, Emiliano & Casella, Francesco & Haglind, Fredrik & Colonna, Piero, 2014. "Design methodology for flexible energy conversion systems accounting for dynamic performance," Energy, Elsevier, vol. 68(C), pages 667-679.
    20. Fang, Jianhao & Hu, Weifei & Liu, Zhenyu & Chen, Weiyi & Tan, Jianrong & Jiang, Zhiyu & Verma, Amrit Shankar, 2022. "Wind turbine rotor speed design optimization considering rain erosion based on deep reinforcement learning," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    21. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    22. Kristin Linnerud & Torben K. Mideksa & Gunnar S. Eskeland, 2011. "The Impact of Climate Change on Nuclear Power Supply," The Energy Journal, International Association for Energy Economics, vol. 0(Number 1), pages 149-168.
    23. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    24. Zio, E., 2018. "The future of risk assessment," Reliability Engineering and System Safety, Elsevier, vol. 177(C), pages 176-190.
    25. Rockafellar, R. Tyrrell & Uryasev, Stanislav, 2002. "Conditional value-at-risk for general loss distributions," Journal of Banking & Finance, Elsevier, vol. 26(7), pages 1443-1471, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    2. Wang, Wei & Cova, Gregorio & Zio, Enrico, 2022. "A clustering-based framework for searching vulnerabilities in the operation dynamics of Cyber-Physical Energy Systems," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    3. Xiong, Jiawei & Zhou, Jian & Ma, Yizhong & Zhang, Fengxia & Lin, Chenglong, 2023. "Adaptive deep learning-based remaining useful life prediction framework for systems with multiple failure patterns," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    4. Pinciroli, Luca & Baraldi, Piero & Ballabio, Guido & Compare, Michele & Zio, Enrico, 2022. "Optimization of the Operation and Maintenance of renewable energy systems by Deep Reinforcement Learning," Renewable Energy, Elsevier, vol. 183(C), pages 752-763.
    5. Saleh, Ali & Chiachío, Manuel & Salas, Juan Fernández & Kolios, Athanasios, 2023. "Self-adaptive optimized maintenance of offshore wind turbines by intelligent Petri nets," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    6. Najafi, Seyedvahid & Lee, Chi-Guhn, 2023. "A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    7. Wei Wang & Francesco Di Maio & Enrico Zio, 2019. "Adversarial Risk Analysis to Allocate Optimal Defense Resources for Protecting Cyber–Physical Systems from Cyber Attacks," Risk Analysis, John Wiley & Sons, vol. 39(12), pages 2766-2785, December.
    8. Zhang, Jiusi & Li, Xiang & Tian, Jilun & Luo, Hao & Yin, Shen, 2023. "An integrated multi-head dual sparse self-attention network for remaining useful life prediction," Reliability Engineering and System Safety, Elsevier, vol. 233(C).
    9. Costa, Nahuel & Sánchez, Luciano, 2022. "Variational encoding approach for interpretable assessment of remaining useful life estimation," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    10. Wang, Chu & Dou, Manfeng & Li, Zhongliang & Outbib, Rachid & Zhao, Dongdong & Zuo, Jian & Wang, Yuanlin & Liang, Bin & Wang, Peng, 2023. "Data-driven prognostics based on time-frequency analysis and symbolic recurrent neural network for fuel cells under dynamic load," Reliability Engineering and System Safety, Elsevier, vol. 233(C).
    11. Mancuso, A. & Compare, M. & Salo, A. & Zio, E., 2021. "Optimal Prognostics and Health Management-driven inspection and maintenance strategies for industrial systems," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    12. Compare, Michele & Antonello, Federico & Pinciroli, Luca & Zio, Enrico, 2022. "A general model for life-cycle cost analysis of Condition-Based Maintenance enabled by PHM capabilities," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    13. Huang, Zhifu & Yang, Yang & Hu, Yawei & Ding, Xiang & Li, Xuanlin & Liu, Yongbin, 2023. "Attention-augmented recalibrated and compensatory network for machine remaining useful life prediction," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    14. Bolbot, Victor & Theotokatos, Gerasimos & Bujorianu, Luminita Manuela & Boulougouris, Evangelos & Vassalos, Dracos, 2019. "Vulnerabilities and safety assurance methods in Cyber-Physical Systems: A comprehensive review," Reliability Engineering and System Safety, Elsevier, vol. 182(C), pages 179-193.
    15. Ifaei, Pouya & Nazari-Heris, Morteza & Tayerani Charmchi, Amir Saman & Asadi, Somayeh & Yoo, ChangKyoo, 2023. "Sustainable energies and machine learning: An organized review of recent applications and challenges," Energy, Elsevier, vol. 266(C).
    16. Morato, P.G. & Andriotis, C.P. & Papakonstantinou, K.G. & Rigo, P., 2023. "Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    17. Terje Aven, 2019. "The Call for a Shift from Risk to Resilience: What Does it Mean?," Risk Analysis, John Wiley & Sons, vol. 39(6), pages 1196-1203, June.
    18. Tohme, Tony & Vanslette, Kevin & Youcef-Toumi, Kamal, 2020. "A generalized Bayesian approach to model calibration," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    19. Lu, Jing & Meng, Yucan & Timmermans, Harry & Zhang, Anming, 2021. "Modeling hesitancy in airport choice: A comparison of discrete choice and machine learning methods," Transportation Research Part A: Policy and Practice, Elsevier, vol. 147(C), pages 230-250.
    20. Vega, Manuel A. & Hu, Zhen & Fillmore, Travis B. & Smith, Matthew D. & Todd, Michael D., 2021. "A Novel Framework for Integration of Abstracted Inspection Data and Structural Health Monitoring for Damage Prognosis of Miter Gates," Reliability Engineering and System Safety, Elsevier, vol. 211(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:235:y:2023:i:c:s0951832023001461. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.