IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v328y2025ics036054422502105x.html
   My bibliography  Save this article

Off-road hybrid electric vehicle energy management strategy using multi-agent soft actor-critic with collaborative-independent algorithm

Author

Listed:
  • Liu, Hui
  • You, Congwen
  • Han, Lijin
  • Yang, Ningkang
  • Liu, Baoshuai

Abstract

Hybrid electric vehicles (HEVs) reduce carbon emissions and save energy, and hybrid energy storage system (HESS) consist of a battery and a supercapacitor which has high energy density and high power density. The HEV equipped with HESS performs better in off-road conditions than single energy storage source. However, its energy management requires multiple input and multiple output (MIMO) control. In this paper, a multi-agent soft actor-critic (MASAC) based energy management strategy (EMS) is proposed to solve the multi-objective optimizing problem considering fuel economy, maintaining state of charge (SOC) and reducing battery state of health (SOH) decay. MASAC based EMS has two advantages: 1) it decomposed the search space into two subspaces, improving the learning efficiency. 2) a novel collaborative-independent algorithm is proposed to allocate rewards among agents, thereby improving the learning stability. Thus, the optimal actions are efficiently and collaboratively learned by two agents, engine agent and HESS agent, showing better performance in multi-objective optimization. In the simulation, the proposed EMS is compared with dynamic programming (DP) and soft actor-critic (SAC) in both off-road driving cycle and standard driving cycles. Simulation results show that the proposed collaborative-independent algorithm enhances the learning efficiency and learning stability of MASAC, while improving the real-time performance of EMS. In off-road conditions, the equivalent fuel consumption of MASAC is slightly better than that of DP. The SOH decay of MASAC is only 20 % higher than DP, significantly outperforming SAC. Furthermore, MASAC demonstrates superior performance in three standard working cycles when compared with SAC.

Suggested Citation

  • Liu, Hui & You, Congwen & Han, Lijin & Yang, Ningkang & Liu, Baoshuai, 2025. "Off-road hybrid electric vehicle energy management strategy using multi-agent soft actor-critic with collaborative-independent algorithm," Energy, Elsevier, vol. 328(C).
    • Handle: RePEc:eee:energy:v:328:y:2025:i:c:s036054422502105x
    DOI: 10.1016/j.energy.2025.136463
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S036054422502105X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2025.136463?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    2. Bo, Lin & Han, Lijin & Xiang, Changle & Liu, Hui & Ma, Tian, 2022. "A Q-learning fuzzy inference system based online energy management strategy for off-road hybrid electric vehicles," Energy, Elsevier, vol. 252(C).
    3. Hui, Sun & Lifu, Yang & Junqing, Jing, 2010. "Hydraulic/electric synergy system (HESS) design for heavy hybrid vehicles," Energy, Elsevier, vol. 35(12), pages 5328-5335.
    4. Huang, Ruchen & He, Hongwen & Su, Qicong, 2024. "Towards a fossil-free urban transport system: An intelligent cross-type transferable energy management framework based on deep transfer reinforcement learning," Applied Energy, Elsevier, vol. 363(C).
    5. Wang, Yong & Wu, Yuankai & Tang, Yingjuan & Li, Qin & He, Hongwen, 2023. "Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 332(C).
    6. Xiong, Rui & Duan, Yanzhou & Cao, Jiayi & Yu, Quanqing, 2018. "Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle," Applied Energy, Elsevier, vol. 217(C), pages 153-165.
    7. da Silva, Samuel Filgueira & Eckert, Jony Javorski & Corrêa, Fernanda Cristina & Silva, Fabrício Leonardo & Silva, Ludmila C.A. & Dedini, Franco Giuseppe, 2022. "Dual HESS electric vehicle powertrain design and fuzzy control based on multi-objective optimization to increase driving range and battery life cycle," Applied Energy, Elsevier, vol. 324(C).
    8. Han, Lijin & You, Congwen & Yang, Ningkang & Liu, Hui & Chen, Ke & Xiang, Changle, 2024. "Adaptive real-time energy management strategy using heuristic search for off-road hybrid electric vehicles," Energy, Elsevier, vol. 304(C).
    9. Yang, Ningkang & Han, Lijin & Bo, Lin & Liu, Baoshuai & Chen, Xiuqi & Liu, Hui & Xiang, Changle, 2023. "Real-time adaptive energy management for off-road hybrid electric vehicles based on decision-time planning," Energy, Elsevier, vol. 282(C).
    10. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    11. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fan Wang & Yina Hong & Xiaohuan Zhao, 2025. "Research and Comparative Analysis of Energy Management Strategies for Hybrid Electric Vehicles: A Review," Energies, MDPI, vol. 18(11), pages 1-28, May.
    2. Huang, Ruchen & He, Hongwen & Su, Qicong, 2024. "Smart energy management for hybrid electric bus via improved soft actor-critic algorithm in a heuristic learning framework," Energy, Elsevier, vol. 309(C).
    3. Huang, Ruchen & He, Hongwen & Su, Qicong & Härtl, Martin & Jaensch, Malte, 2024. "Enabling cross-type full-knowledge transferable energy management for hybrid electric vehicles via deep transfer reinforcement learning," Energy, Elsevier, vol. 305(C).
    4. Tang, Tianfeng & Peng, Qianlong & Shi, Qing & Peng, Qingguo & Zhao, Jin & Chen, Chaoyi & Wang, Guangwei, 2024. "Energy management of fuel cell hybrid electric bus in mountainous regions: A deep reinforcement learning approach considering terrain characteristics," Energy, Elsevier, vol. 311(C).
    5. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    6. Han, Lijin & You, Congwen & Yang, Ningkang & Liu, Hui & Chen, Ke & Xiang, Changle, 2024. "Adaptive real-time energy management strategy using heuristic search for off-road hybrid electric vehicles," Energy, Elsevier, vol. 304(C).
    7. Liu, Weirong & Yao, Pengfei & Wu, Yue & Duan, Lijun & Li, Heng & Peng, Jun, 2025. "Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system," Applied Energy, Elsevier, vol. 378(PA).
    8. Wang, Zhong & Zhao, Yahui & Zhang, Yahui & Tian, Yang & Jiao, Xiaohong, 2024. "Safe off-policy reinforcement learning on eco-driving for a P2-P3 hybrid electric truck," Energy, Elsevier, vol. 313(C).
    9. Dai, Churong & Zuo, Wei & Li, Qingqing & Zhou, Kun & Huang, Yuhan & Zhang, Guangde & E, Jiaqiang, 2024. "Energy conversion efficiency improvement studies on the hydrogen-fueled micro planar combustor with multi-baffles for thermophotovoltaic applications," Energy, Elsevier, vol. 313(C).
    10. Lei, Nuo & Zhang, Hao & Hu, Jingjing & Hu, Zunyan & Wang, Zhi, 2025. "Sim-to-real design and development of reinforcement learning-based energy management strategies for fuel cell electric vehicles," Applied Energy, Elsevier, vol. 393(C).
    11. Fang, Shuo & Hu, Shuangxi & Liu, Yuntao & Zhao, Chunhui & Wang, Ying, 2025. "Power management unit with maximum-efficiency-point-tracking to enhance the efficiency of micro DMFC stack," Energy, Elsevier, vol. 315(C).
    12. Chen, Fujun & Wang, Bowen & Ni, Meng & Gong, Zhichao & Jiao, Kui, 2024. "Online energy management strategy for ammonia-hydrogen hybrid electric vehicles harnessing deep reinforcement learning," Energy, Elsevier, vol. 301(C).
    13. Tan, Yingqi & Xu, Jingyi & Ma, Junyi & Li, Zirui & Chen, Huiyan & Xi, Junqiang & Liu, Haiou, 2024. "A transferable perception-guided EMS for series hybrid electric unmanned tracked vehicles," Energy, Elsevier, vol. 306(C).
    14. Li, Hao & Zhou, Xingfei & Hu, Ziyang & Zhang, Houcheng, 2025. "Waste heat recycling from phosphoric acid fuel cells for desalination with hydrophilic modified tubular stills: Performance prediction and regulation," Energy, Elsevier, vol. 317(C).
    15. Liu, Zemin Eitan & Li, Yong & Zhou, Quan & Shuai, Bin & Hua, Min & Xu, Hongming & Xu, Lubing & Tan, Guikun & Li, Yanfei, 2025. "Real-time energy management for HEV combining naturalistic driving data and deep reinforcement learning with high generalization," Applied Energy, Elsevier, vol. 377(PA).
    16. Chen, Bin & Wang, Miaoben & Hu, Lin & He, Guo & Yan, Haoyang & Wen, Xinji & Du, Ronghua, 2024. "Data-driven Koopman model predictive control for hybrid energy storage system of electric vehicles under vehicle-following scenarios," Applied Energy, Elsevier, vol. 365(C).
    17. Huang, Ruchen & He, Hongwen & Su, Qicong, 2024. "An intelligent full-knowledge transferable collaborative eco-driving framework based on improved soft actor-critic algorithm," Applied Energy, Elsevier, vol. 375(C).
    18. Niu, Zegong & He, Hongwen, 2024. "A data-driven solution for intelligent power allocation of connected hybrid electric vehicles inspired by offline deep reinforcement learning in V2X scenario," Applied Energy, Elsevier, vol. 372(C).
    19. He, Hongwen & Su, Qicong & Huang, Ruchen & Niu, Zegong, 2024. "Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm," Energy, Elsevier, vol. 294(C).
    20. Li, Jianwei & Liu, Jie & Yang, Qingqing & Wang, Tianci & He, Hongwen & Wang, Hanxiao & Sun, Fengchun, 2025. "Reinforcement learning based energy management for fuel cell hybrid electric vehicles: A comprehensive review on decision process reformulation and strategy implementation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 213(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:328:y:2025:i:c:s036054422502105x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.