IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v283y2023ics0360544223019308.html
   My bibliography  Save this article

An energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating conditions

Author

Listed:
  • Chang, Chengcheng
  • Zhao, Wanzhong
  • Wang, Chunyan
  • Luan, Zhongkai

Abstract

To improve the driving efficiency of hybrid power vehicle, an energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating vehicle driving conditions is proposed. Firstly, the kinematics segments are self-generated based on the Wasserstein generative adversarial network. The generator network G is used to generate kinematics segments. The discriminator network D is used to judge the credibility of the generated kinematics segments with the Wasserstein distance. The speed distribution characteristics of the training conditions and verification conditions established based on the self-generated segments are verified. Afterward, a multi-agent algorithm based on twin delayed deep deterministic policy gradient algorithm for hybrid systems is proposed by introducing centralized training with decentralized execution framework. The engine and a motor are used as two independent agents respectively. Different reward functions are designed based on training objectives to establish a mutually beneficial relationship of cooperation-restraint between the two agents. A driving mode constraint is designed in the environment to improve sample utilization. Finally, the simulation results demonstrate that our method can achieve better performance compared with other existing works.

Suggested Citation

  • Chang, Chengcheng & Zhao, Wanzhong & Wang, Chunyan & Luan, Zhongkai, 2023. "An energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating conditions," Energy, Elsevier, vol. 283(C).
    • Handle: RePEc:eee:energy:v:283:y:2023:i:c:s0360544223019308
    DOI: 10.1016/j.energy.2023.128536
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223019308
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.128536?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Trovão, João P. & Pereirinha, Paulo G. & Jorge, Humberto M. & Antunes, Carlos Henggeler, 2013. "A multi-level energy management system for multi-source electric vehicles – An integrated rule-based meta-heuristic approach," Applied Energy, Elsevier, vol. 105(C), pages 304-318.
    2. Peng, Jiankun & He, Hongwen & Xiong, Rui, 2017. "Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming," Applied Energy, Elsevier, vol. 185(P2), pages 1633-1643.
    3. Min, Dehao & Song, Zhen & Chen, Huicui & Wang, Tianxiang & Zhang, Tong, 2022. "Genetic algorithm optimized neural network based fuel cell hybrid electric vehicle energy management strategy under start-stop condition," Applied Energy, Elsevier, vol. 306(PB).
    4. Sun, Wenjing & Zou, Yuan & Zhang, Xudong & Guo, Ningyuan & Zhang, Bin & Du, Guodong, 2022. "High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning," Energy, Elsevier, vol. 258(C).
    5. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    6. Tang, Wenbin & Wang, Yaqian & Jiao, Xiaohong & Ren, Lina, 2023. "Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios," Energy, Elsevier, vol. 265(C).
    7. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun, 2022. "Coordinated load frequency control of multi-area integrated energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 306(PA).
    8. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    9. Zhang, Fengqi & Xiao, Lehua & Coskun, Serdar & Pang, Hui & Xie, Shaobo & Liu, Kailong & Cui, Yahui, 2023. "Comparative study of energy management in parallel hybrid electric vehicles considering battery ageing," Energy, Elsevier, vol. 264(C).
    10. Li, Yuecheng & He, Hongwen & Khajepour, Amir & Wang, Hong & Peng, Jiankun, 2019. "Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information," Applied Energy, Elsevier, vol. 255(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Murshed, Shabab & Nibir, Abu Shaikh & Razzaque, Md. Abdur & Roy, Palash & Elhendi, Ahmed Zohier & Hassan, Md. Rafiul & Hassan, Mohammad Mehedi, 2024. "Weighted fair energy transfer in a UAV network: A multi-agent deep reinforcement learning approach," Energy, Elsevier, vol. 292(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Yonggang & Liu, Junjun & Zhang, Yuanjian & Wu, Yitao & Chen, Zheng & Ye, Ming, 2020. "Rule learning based energy management strategy of fuel cell hybrid vehicles considering multi-objective optimization," Energy, Elsevier, vol. 207(C).
    2. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    3. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    4. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    5. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    6. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    7. Kong, Yan & Xu, Nan & Zhang, Yuanjian & Sui, Yan & Ju, Hao & Liu, Heng & Xu, Zhe, 2021. "Acquisition of full-factor trip information for global optimization energy management in multi-energy source vehicles and the measure of the amount of information to be transmitted," Energy, Elsevier, vol. 236(C).
    8. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    9. Wu, Yitao & Zhang, Yuanjian & Li, Guang & Shen, Jiangwei & Chen, Zheng & Liu, Yonggang, 2020. "A predictive energy management strategy for multi-mode plug-in hybrid electric vehicles based on multi neural networks," Energy, Elsevier, vol. 208(C).
    10. Tao, Fazhan & Fu, Zhigao & Gong, Huixian & Ji, Baofeng & Zhou, Yao, 2023. "Twin delayed deep deterministic policy gradient based energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles considering predicted terrain information," Energy, Elsevier, vol. 283(C).
    11. Zhang, Yahui & Wang, Zimeng & Tian, Yang & Wang, Zhong & Kang, Mingxin & Xie, Fangxi & Wen, Guilin, 2024. "Pre-optimization-assisted deep reinforcement learning-based energy management strategy for a series–parallel hybrid electric truck," Energy, Elsevier, vol. 302(C).
    12. Zhang, Wei & Wang, Jixin & Xu, Zhenyu & Shen, Yuying & Gao, Guangzong, 2022. "A generalized energy management framework for hybrid construction vehicles via model-based reinforcement learning," Energy, Elsevier, vol. 260(C).
    13. Xiong, Rui & Duan, Yanzhou & Cao, Jiayi & Yu, Quanqing, 2018. "Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle," Applied Energy, Elsevier, vol. 217(C), pages 153-165.
    14. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    15. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    16. Anselma, Pier Giuseppe, 2022. "Computationally efficient evaluation of fuel and electrical energy economy of plug-in hybrid electric vehicles with smooth driving constraints," Applied Energy, Elsevier, vol. 307(C).
    17. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    18. Yang, Ningkang & Han, Lijin & Bo, Lin & Liu, Baoshuai & Chen, Xiuqi & Liu, Hui & Xiang, Changle, 2023. "Real-time adaptive energy management for off-road hybrid electric vehicles based on decision-time planning," Energy, Elsevier, vol. 282(C).
    19. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    20. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:283:y:2023:i:c:s0360544223019308. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.