IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v275y2020ics0306261920307704.html
   My bibliography  Save this article

Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships

Author

Listed:
  • Wu, Peng
  • Partridge, Julius
  • Bucknall, Richard

Abstract

Hybrid fuel cell and battery propulsion systems have the potential to offer improved emission performance for coastal ships with access to H2 replenishment and battery charging infrastructures in ports. However, such systems could be constrained by high power source degradation and energy costs. Cost-effective energy management strategies are essential for such hybrid systems to mitigate the high costs. This article presents a Double Q reinforcement learning based energy management system for such systems to achieve near-optimal average voyage cost. The Double Q agent is trained using stochastic power profiles collected from continuous monitoring of a passenger ferry, using a plug-in hybrid fuel cell and battery propulsion system model. The energy management strategies generated by the agent were validated using another test dataset collected over a different period. The proposed methodology provides a novel approach to optimal use hybrid fuel cell and battery propulsion systems for ships. The results show that without prior knowledge of future power demands, the strategies can achieve near-optimal cost performance (96.9%) compared to those derived from using dynamic programming with the equivalent state space resolution.

Suggested Citation

  • Wu, Peng & Partridge, Julius & Bucknall, Richard, 2020. "Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships," Applied Energy, Elsevier, vol. 275(C).
    • Handle: RePEc:eee:appene:v:275:y:2020:i:c:s0306261920307704
    DOI: 10.1016/j.apenergy.2020.115258
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261920307704
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2020.115258?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sulaiman, N. & Hannan, M.A. & Mohamed, A. & Majlan, E.H. & Wan Daud, W.R., 2015. "A review on energy management system for fuel cell hybrid electric vehicle: Issues and challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 52(C), pages 802-814.
    2. Jingang Han & Jean-Frederic Charpentier & Tianhao Tang, 2014. "An Energy Management System of a Fuel Cell/Battery Hybrid Boat," Energies, MDPI, vol. 7(5), pages 1-22, April.
    3. Zheng, Fangdan & Xing, Yinjiao & Jiang, Jiuchun & Sun, Bingxiang & Kim, Jonghoon & Pecht, Michael, 2016. "Influence of different open circuit voltage tests on state of charge online estimation for lithium-ion batteries," Applied Energy, Elsevier, vol. 183(C), pages 513-525.
    4. Chen, Huicui & Pei, Pucheng & Song, Mancun, 2015. "Lifetime prediction and the economic lifetime of Proton Exchange Membrane fuel cells," Applied Energy, Elsevier, vol. 142(C), pages 154-163.
    5. Omar, Noshin & Monem, Mohamed Abdel & Firouz, Yousef & Salminen, Justin & Smekens, Jelle & Hegazy, Omar & Gaulous, Hamid & Mulder, Grietus & Van den Bossche, Peter & Coosemans, Thierry & Van Mierlo, J, 2014. "Lithium iron phosphate based battery – Assessment of the aging parameters and development of cycle life model," Applied Energy, Elsevier, vol. 113(C), pages 1575-1585.
    6. Hu, Xiaosong & Johannesson, Lars & Murgovski, Nikolce & Egardt, Bo, 2015. "Longevity-conscious dimensioning and power management of the hybrid energy storage system in a fuel cell hybrid electric bus," Applied Energy, Elsevier, vol. 137(C), pages 913-924.
    7. Ximing Wang & Hongwen He & Fengchun Sun & Jieli Zhang, 2015. "Application Study on the Dynamic Programming Algorithm for Energy Management of Plug-in Hybrid Electric Vehicles," Energies, MDPI, vol. 8(4), pages 1-20, April.
    8. Wang, Yujie & Sun, Zhendong & Li, Xiyun & Yang, Xiaoyu & Chen, Zonghai, 2019. "A comparative study of power allocation strategies used in fuel cell and ultracapacitor hybrid systems," Energy, Elsevier, vol. 189(C).
    9. Rouholamini, Mehdi & Mohammadian, Mohsen, 2016. "Heuristic-based power management of a grid-connected hybrid energy system combined with hydrogen storage," Renewable Energy, Elsevier, vol. 96(PA), pages 354-365.
    10. Shuxian Li & Minghui Hu & Changchao Gong & Sen Zhan & Datong Qin, 2018. "Energy Management Strategy for Hybrid Electric Vehicle Based on Driving Condition Identification Using KGA-Means," Energies, MDPI, vol. 11(6), pages 1-16, June.
    11. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    12. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    13. Shen, Peihong & Zhao, Zhiguo & Zhan, Xiaowen & Li, Jingwei & Guo, Qiuyi, 2018. "Optimal energy management strategy for a plug-in hybrid electric commercial vehicle based on velocity prediction," Energy, Elsevier, vol. 155(C), pages 838-852.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Macias, A. & Kandidayeni, M. & Boulon, L. & Trovão, J.P., 2021. "Fuel cell-supercapacitor topologies benchmark for a three-wheel electric vehicle powertrain," Energy, Elsevier, vol. 224(C).
    2. Kun He & Dongchen Qin & Jiangyi Chen & Tingting Wang & Hongxia Wu & Peizhuo Wang, 2023. "Adaptive Equivalent Consumption Minimization Strategy for Fuel Cell Buses Based on Driving Style Recognition," Sustainability, MDPI, vol. 15(10), pages 1-17, May.
    3. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    4. Lin, Xinyou & Xu, Xinhao & Wang, Zhaorui, 2022. "Deep Q-learning network based trip pattern adaptive battery longevity-conscious strategy of plug-in fuel cell hybrid electric vehicle," Applied Energy, Elsevier, vol. 321(C).
    5. Elkholy, M.H. & Metwally, Hamid & Farahat, M.A. & Senjyu, Tomonobu & Elsayed Lotfy, Mohammed, 2022. "Smart centralized energy management system for autonomous microgrid using FPGA," Applied Energy, Elsevier, vol. 317(C).
    6. Yang, Zirong & Jiao, Kui & Wu, Kangcheng & Shi, Weilong & Jiang, Shangfeng & Zhang, Longhai & Du, Qing, 2021. "Numerical investigations of assisted heating cold start strategies for proton exchange membrane fuel cell systems," Energy, Elsevier, vol. 222(C).
    7. Perčić, Maja & Frković, Lovro & Pukšec, Tomislav & Ćosić, Boris & Li, Oi Lun & Vladimir, Nikola, 2022. "Life-cycle assessment and life-cycle cost assessment of power batteries for all-electric vessels for short-sea navigation," Energy, Elsevier, vol. 251(C).
    8. Chiara Dall’Armi & Davide Pivetta & Rodolfo Taccani, 2021. "Health-Conscious Optimization of Long-Term Operation for Hybrid PEMFC Ship Propulsion Systems," Energies, MDPI, vol. 14(13), pages 1-20, June.
    9. Li, Yapeng & Wang, Feng & Tang, Xiaolin & Hu, Xiaosong & Lin, Xianke, 2022. "Convex optimization-based predictive and bi-level energy management for plug-in hybrid electric vehicles," Energy, Elsevier, vol. 257(C).
    10. Chiara Dall’Armi & Davide Pivetta & Rodolfo Taccani, 2023. "Hybrid PEM Fuel Cell Power Plants Fuelled by Hydrogen for Improving Sustainability in Shipping: State of the Art and Review on Active Projects," Energies, MDPI, vol. 16(4), pages 1-34, February.
    11. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    12. Touzani, Samir & Prakash, Anand Krishnan & Wang, Zhe & Agarwal, Shreya & Pritoni, Marco & Kiran, Mariam & Brown, Richard & Granderson, Jessica, 2021. "Controlling distributed energy resources via deep reinforcement learning for load flexibility and energy efficiency," Applied Energy, Elsevier, vol. 304(C).
    13. Tsoumpris, Charalampos & Theotokatos, Gerasimos, 2023. "A decision-making approach for the health-aware energy management of ship hybrid power plants," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    14. Xie, Peilin & Tan, Sen & Bazmohammadi, Najmeh & Guerrero, Josep. M. & Vasquez, Juan. C. & Alcala, Jose Matas & Carreño, Jorge El Mariachet, 2022. "A distributed real-time power management scheme for shipboard zonal multi-microgrid system," Applied Energy, Elsevier, vol. 317(C).
    15. Chen, Chunyu & Cui, Mingjian & Fang, Xin & Ren, Bixing & Chen, Yang, 2020. "Load altering attack-tolerant defense strategy for load frequency control system," Applied Energy, Elsevier, vol. 280(C).
    16. Maja Perčić & Nikola Vladimir & Marija Koričan, 2021. "Electrification of Inland Waterway Ships Considering Power System Lifetime Emissions and Costs," Energies, MDPI, vol. 14(21), pages 1-25, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    2. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    3. Li, Weihan & Cui, Han & Nemeth, Thomas & Jansen, Jonathan & Ünlübayir, Cem & Wei, Zhongbao & Feng, Xuning & Han, Xuebing & Ouyang, Minggao & Dai, Haifeng & Wei, Xuezhe & Sauer, Dirk Uwe, 2021. "Cloud-based health-conscious energy management of hybrid battery systems in electric vehicles with deep reinforcement learning," Applied Energy, Elsevier, vol. 293(C).
    4. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    5. da Silva, Samuel Filgueira & Eckert, Jony Javorski & Corrêa, Fernanda Cristina & Silva, Fabrício Leonardo & Silva, Ludmila C.A. & Dedini, Franco Giuseppe, 2022. "Dual HESS electric vehicle powertrain design and fuzzy control based on multi-objective optimization to increase driving range and battery life cycle," Applied Energy, Elsevier, vol. 324(C).
    6. Zhuang, Weichao & Li (Eben), Shengbo & Zhang, Xiaowu & Kum, Dongsuk & Song, Ziyou & Yin, Guodong & Ju, Fei, 2020. "A survey of powertrain configuration studies on hybrid electric vehicles," Applied Energy, Elsevier, vol. 262(C).
    7. Hou, Shengyan & Yin, Hai & Xu, Fuguo & Benjamín, Pla & Gao, Jinwu & Chen, Hong, 2023. "Multihorizon predictive energy optimization and lifetime management for connected fuel cell electric vehicles," Energy, Elsevier, vol. 266(C).
    8. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    9. Wang, Yue & Li, Keqiang & Zeng, Xiaohua & Gao, Bolin & Hong, Jichao, 2023. "Investigation of novel intelligent energy management strategies for connected HEB considering global planning of fixed-route information," Energy, Elsevier, vol. 263(PB).
    10. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    11. Zhuang, Weichao & Ye, Jianwei & Song, Ziyou & Yin, Guodong & Li, Guangmin, 2020. "Comparison of semi-active hybrid battery system configurations for electric taxis application," Applied Energy, Elsevier, vol. 259(C).
    12. Tran, Dai-Duong & Vafaeipour, Majid & El Baghdadi, Mohamed & Barrero, Ricardo & Van Mierlo, Joeri & Hegazy, Omar, 2020. "Thorough state-of-the-art analysis of electric and hybrid vehicle powertrains: Topologies and integrated energy management strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 119(C).
    13. Nie, Zhigen & Jia, Yuan & Wang, Wanqiong & Chen, Zheng & Outbib, Rachid, 2022. "Co-optimization of speed planning and energy management for intelligent fuel cell hybrid vehicle considering complex traffic conditions," Energy, Elsevier, vol. 247(C).
    14. Li, Tianyu & Liu, Huiying & Wang, Hui & Yao, Yongming, 2020. "Hierarchical predictive control-based economic energy management for fuel cell hybrid construction vehicles," Energy, Elsevier, vol. 198(C).
    15. Juhui Gim & Minsu Kim & Changsun Ahn, 2022. "Energy Management Control Strategy for Saving Trip Costs of Fuel Cell/Battery Electric Vehicles," Energies, MDPI, vol. 15(6), pages 1-15, March.
    16. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    17. Sanghyun Yun & Jinwon Yun & Jaeyoung Han, 2023. "Development of a 470-Horsepower Fuel Cell–Battery Hybrid Xcient Dynamic Model Using Simscape TM," Energies, MDPI, vol. 16(24), pages 1-22, December.
    18. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    19. Wang, Yujie & Sun, Zhendong & Chen, Zonghai, 2019. "Energy management strategy for battery/supercapacitor/fuel cell hybrid source vehicles based on finite state machine," Applied Energy, Elsevier, vol. 254(C).
    20. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:275:y:2020:i:c:s0306261920307704. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.