IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v332y2023ics0306261922018207.html
   My bibliography  Save this article

Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning

Author

Listed:
  • Wang, Yong
  • Wu, Yuankai
  • Tang, Yingjuan
  • Li, Qin
  • He, Hongwen

Abstract

The advanced cruise control system has expanded the energy-saving potential of the hybrid electric vehicle (HEV). Despite this, most energy-saving researches for HEV either only optimize the energy management strategy (EMS) or integrate eco-driving through a hierarchically optimized assumption that optimizes EMS and eco-driving separately. Such kinds of approaches may lead to sub-optimal results. To fill this gap, we design a multi-agent reinforcement learning (MARL) based optimal energy-saving strategy for HEV, achieving a cooperative control on the powertrain and car-following behaviors to minimize the energy consumption and keep a safe following distance simultaneously. Specifically, a plug-in HEV model is regarded as the research object in this paper. Firstly, the HEV energy management problem in the car-following scenario is decomposed into a multi-agent cooperative task into two subtasks, each of which can conduct interactive learning through cooperative optimization. Secondly, the energy-saving strategy is designed, called the independent soft actor–critic, which consists of a car-following agent and an energy management agent. Finally, the performance of velocity tracking and energy-saving are validated under different driving cycles. In comparison to the state-of-the-art hierarchical model predictive control (MPC) strategy, the proposed MARL method can reduce fuel consumption by 15.8% while ensuring safety and comfort.

Suggested Citation

  • Wang, Yong & Wu, Yuankai & Tang, Yingjuan & Li, Qin & He, Hongwen, 2023. "Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 332(C).
    • Handle: RePEc:eee:appene:v:332:y:2023:i:c:s0306261922018207
    DOI: 10.1016/j.apenergy.2022.120563
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922018207
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.120563?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhou, Yang & Ahn, Soyoung, 2019. "Robust local and string stability for a decentralized car following control strategy for connected automated vehicles," Transportation Research Part B: Methodological, Elsevier, vol. 125(C), pages 175-196.
    2. Tian, He & Li, Shengbo Eben & Wang, Xu & Huang, Yong & Tian, Guangyu, 2018. "Data-driven hierarchical control for online energy management of plug-in hybrid electric city bus," Energy, Elsevier, vol. 142(C), pages 55-67.
    3. Bellocchi, Sara & Klöckner, Kai & Manno, Michele & Noussan, Michel & Vellini, Michela, 2019. "On the role of electric vehicles towards low-carbon energy systems: Italy and Germany in comparison," Applied Energy, Elsevier, vol. 255(C).
    4. Ibrahim, Amier & Jiang, Fangming, 2021. "The electric vehicle energy management: An overview of the energy system and related modeling and simulation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 144(C).
    5. Quan, Shengwei & Wang, Ya-Xiong & Xiao, Xuelian & He, Hongwen & Sun, Fengchun, 2021. "Real-time energy management for fuel cell electric vehicle using speed prediction-based model predictive control considering performance degradation," Applied Energy, Elsevier, vol. 304(C).
    6. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    7. Zhang, Jian & Tang, Tie-Qiao & Yan, Yadan & Qu, Xiaobo, 2021. "Eco-driving control for connected and automated electric vehicles at signalized intersections with wireless charging," Applied Energy, Elsevier, vol. 282(PA).
    8. Gao, Zhiming & LaClair, Tim & Ou, Shiqi & Huff, Shean & Wu, Guoyuan & Hao, Peng & Boriboonsomsin, Kanok & Barth, Matthew, 2019. "Evaluation of electric vehicle component performance over eco-driving cycles," Energy, Elsevier, vol. 172(C), pages 823-839.
    9. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    10. Ruan, Shumin & Ma, Yue & Yang, Ningkang & Xiang, Changle & Li, Xunming, 2022. "Real-time energy-saving control for HEVs in car-following scenario with a double explicit MPC approach," Energy, Elsevier, vol. 247(C).
    11. Qu, Xiaobo & Yu, Yang & Zhou, Mofan & Lin, Chin-Teng & Wang, Xiangyu, 2020. "Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: A reinforcement learning based approach," Applied Energy, Elsevier, vol. 257(C).
    12. Hongwen, He & Jinquan, Guo & Jiankun, Peng & Huachun, Tan & Chao, Sun, 2018. "Real-time global driving cycle construction and the application to economy driving pro system in plug-in hybrid electric vehicles," Energy, Elsevier, vol. 152(C), pages 95-107.
    13. Lee, Heeyun & Kim, Kyunghyun & Kim, Namwook & Cha, Suk Won, 2022. "Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning," Applied Energy, Elsevier, vol. 313(C).
    14. Peter R. Wurman & Samuel Barrett & Kenta Kawamoto & James MacGlashan & Kaushik Subramanian & Thomas J. Walsh & Roberto Capobianco & Alisa Devlic & Franziska Eckert & Florian Fuchs & Leilani Gilpin & P, 2022. "Outracing champion Gran Turismo drivers with deep reinforcement learning," Nature, Nature, vol. 602(7896), pages 223-228, February.
    15. Li, Yuecheng & He, Hongwen & Khajepour, Amir & Wang, Hong & Peng, Jiankun, 2019. "Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information," Applied Energy, Elsevier, vol. 255(C).
    16. Huang, Yuhan & Ng, Elvin C.Y. & Zhou, John L. & Surawski, Nic C. & Chan, Edward F.C. & Hong, Guang, 2018. "Eco-driving technology for sustainable road transport: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 93(C), pages 596-609.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    2. Hua, Min & Zhang, Cetengfei & Zhang, Fanggang & Li, Zhi & Yu, Xiaoli & Xu, Hongming & Zhou, Quan, 2023. "Energy management of multi-mode plug-in hybrid electric vehicle using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 348(C).
    3. Li, Jie & Fotouhi, Abbas & Pan, Wenjun & Liu, Yonggang & Zhang, Yuanjian & Chen, Zheng, 2023. "Deep reinforcement learning-based eco-driving control for connected electric vehicles at signalized intersections considering traffic uncertainties," Energy, Elsevier, vol. 279(C).
    4. Hou, Zhuoran & Guo, Jianhua & Chu, Liang & Hu, Jincheng & Chen, Zheng & Zhang, Yuanjian, 2023. "Exploration the route of information integration for vehicle design: A knowledge-enhanced energy management strategy," Energy, Elsevier, vol. 282(C).
    5. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    6. Xu Wang & Ying Huang & Jian Wang, 2023. "Study on Driver-Oriented Energy Management Strategy for Hybrid Heavy-Duty Off-Road Vehicles under Aggressive Transient Operating Condition," Sustainability, MDPI, vol. 15(9), pages 1-25, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    2. Hou, Shengyan & Yin, Hai & Xu, Fuguo & Benjamín, Pla & Gao, Jinwu & Chen, Hong, 2023. "Multihorizon predictive energy optimization and lifetime management for connected fuel cell electric vehicles," Energy, Elsevier, vol. 266(C).
    3. Dong, Haoxuan & Zhuang, Weichao & Chen, Boli & Wang, Yan & Lu, Yanbo & Liu, Ying & Xu, Liwei & Yin, Guodong, 2022. "A comparative study of energy-efficient driving strategy for connected internal combustion engine and electric vehicles at signalized intersections," Applied Energy, Elsevier, vol. 310(C).
    4. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    5. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    6. Zhang, Hanyu & Du, Lili, 2023. "Platoon-centered control for eco-driving at signalized intersection built upon hybrid MPC system, online learning and distributed optimization part I: Modeling and solution algorithm design," Transportation Research Part B: Methodological, Elsevier, vol. 172(C), pages 174-198.
    7. Ahmed, Abdelsalam A. & Ramadan, Haitham S., 2020. "Prototype implementation of advanced electric vehicles drivetrain system: Verification and validation," Applied Energy, Elsevier, vol. 266(C).
    8. Amjad, Muhammad & Farooq-i-Azam, Muhammad & Ni, Qiang & Dong, Mianxiong & Ansari, Ejaz Ahmad, 2022. "Wireless charging systems for electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    9. Zhang, Yuanjian & Liu, Yonggang & Huang, Yanjun & Chen, Zheng & Li, Guang & Hao, Wanming & Cunningham, Geoff & Early, Juliana, 2021. "An optimal control strategy design for plug-in hybrid electric vehicles based on internet of vehicles," Energy, Elsevier, vol. 228(C).
    10. Sun, Xilei & Fu, Jianqin & Yang, Huiyong & Xie, Mingke & Liu, Jingping, 2023. "An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control," Energy, Elsevier, vol. 269(C).
    11. Zhou, Yang & Zhong, Xinzhi & Chen, Qian & Ahn, Soyoung & Jiang, Jiwan & Jafarsalehi, Ghazaleh, 2023. "Data-driven analysis for disturbance amplification in car-following behavior of automated vehicles," Transportation Research Part B: Methodological, Elsevier, vol. 174(C).
    12. Chen, Zheng & Wu, Simin & Shen, Shiquan & Liu, Yonggang & Guo, Fengxiang & Zhang, Yuanjian, 2023. "Co-optimization of velocity planning and energy management for autonomous plug-in hybrid electric vehicles in urban driving scenarios," Energy, Elsevier, vol. 263(PF).
    13. Nie, Zhigen & Jia, Yuan & Wang, Wanqiong & Chen, Zheng & Outbib, Rachid, 2022. "Co-optimization of speed planning and energy management for intelligent fuel cell hybrid vehicle considering complex traffic conditions," Energy, Elsevier, vol. 247(C).
    14. Zhang, Hao & Fan, Qinhao & Liu, Shang & Li, Shengbo Eben & Huang, Jin & Wang, Zhi, 2021. "Hierarchical energy management strategy for plug-in hybrid electric powertrain integrated with dual-mode combustion engine," Applied Energy, Elsevier, vol. 304(C).
    15. Min, Qingyun & Li, Junqiu & Liu, Bo & Li, Jianwei & Sun, Fengchun & Sun, Chao, 2021. "Guided model predictive control for connected vehicles with hybrid energy systems," Energy, Elsevier, vol. 230(C).
    16. Saeed Vasebi & Yeganeh M. Hayeri, 2021. "Collective Driving to Mitigate Climate Change: Collective-Adaptive Cruise Control," Sustainability, MDPI, vol. 13(16), pages 1-30, August.
    17. Xie, Yunkun & Li, Yangyang & Zhao, Zhichao & Dong, Hao & Wang, Shuqian & Liu, Jingping & Guan, Jinhuan & Duan, Xiongbo, 2020. "Microsimulation of electric vehicle energy consumption and driving range," Applied Energy, Elsevier, vol. 267(C).
    18. Yashraj Tripathy & Andrew McGordon & Anup Barai, 2020. "Improving Accessible Capacity Tracking at Low Ambient Temperatures for Range Estimation of Battery Electric Vehicles," Energies, MDPI, vol. 13(8), pages 1-18, April.
    19. Østergaard, P.A. & Lund, H. & Thellufsen, J.Z. & Sorknæs, P. & Mathiesen, B.V., 2022. "Review and validation of EnergyPLAN," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    20. Biyao Wang & Yi Han & Siyu Wang & Di Tian & Mengjiao Cai & Ming Liu & Lujia Wang, 2022. "A Review of Intelligent Connected Vehicle Cooperative Driving Development," Mathematics, MDPI, vol. 10(19), pages 1-31, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:332:y:2023:i:c:s0306261922018207. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.