IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v288y2024ics0360544223031675.html
   My bibliography  Save this article

FlexNet: A warm start method for deep reinforcement learning in hybrid electric vehicle energy management applications

Author

Listed:
  • Wang, Hanchen
  • Arjmandzadeh, Ziba
  • Ye, Yiming
  • Zhang, Jiangfeng
  • Xu, Bin

Abstract

Deep reinforcement learning (DRL) has been widely studied in the energy management of hybrid electric vehicles (HEVs) for its remarkable energy efficiency improvement compared to conventional methods. However, how to alleviate the time consumption of training a stable reinforcement learning agent still needs to be solved in real-world implementation. This study presents a human expert knowledge encoded ‘warm start’ method with the flexibility to change the neural network architecture. The expert knowledge is encoded in a decision tree which then initializes the weights and bias of the DRL neural network. Compared with another fixed architecture warm start method, the proposed FlexNet exhibits improved learning speed by 60.8 % and 88.8 % in action space 50 and 100, respectively. The energy consumption by the proposed FlexNet EMS method is 12.2 % and 6.4 % better than rule-based and equivalent consumption minimization strategy, respectively. This proposed warm start method can reduce learning time and increase energy efficiency in various energy management applications.

Suggested Citation

  • Wang, Hanchen & Arjmandzadeh, Ziba & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2024. "FlexNet: A warm start method for deep reinforcement learning in hybrid electric vehicle energy management applications," Energy, Elsevier, vol. 288(C).
    • Handle: RePEc:eee:energy:v:288:y:2024:i:c:s0360544223031675
    DOI: 10.1016/j.energy.2023.129773
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223031675
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.129773?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Ma, Qingyin & Stachurski, John & Toda, Alexis Akira, 2022. "Unbounded dynamic programming via the Q-transform," Journal of Mathematical Economics, Elsevier, vol. 100(C).
    2. Sun, Wenjing & Zou, Yuan & Zhang, Xudong & Guo, Ningyuan & Zhang, Bin & Du, Guodong, 2022. "High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning," Energy, Elsevier, vol. 258(C).
    3. Pérez, Laura V. & Bossio, Guillermo R. & Moitre, Diego & García, Guillermo O., 2006. "Optimization of power management in an hybrid electric vehicle using dynamic programming," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 73(1), pages 244-254.
    4. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    5. Al-Alawi, Baha M. & Bradley, Thomas H., 2013. "Review of hybrid, plug-in hybrid, and electric vehicle market modeling Studies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 21(C), pages 190-203.
    6. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cai, Xuan & Zhou, Wei & Cui, Zhiyong & Bai, Xuesong & Liu, Fan & Yu, Haiyang & Ren, Yilong, 2024. "An explicit State-of-Charge planning solution for plug-in hybrid electric vehicle based on low-granularity prior-knowledge," Energy, Elsevier, vol. 313(C).
    2. Dawei Zhong & Bolan Liu & Liang Liu & Wenhao Fan & Jingxian Tang, 2025. "Artificial Intelligence Algorithms for Hybrid Electric Powertrain System Control: A Review," Energies, MDPI, vol. 18(8), pages 1-30, April.
    3. Najmi, Aezid-Ul-Hassan & Wahab, Abdul & Prakash, Rohith & Schopen, Oliver & Esch, Thomas & Shabani, Bahman, 2025. "Thermal management of fuel cell-battery electric vehicles: Challenges and solutions," Applied Energy, Elsevier, vol. 387(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Zhang, Dongfang & Sun, Wei & Zou, Yuan & Zhang, Xudong, 2025. "Energy management in HDHEV with dual APUs: Enhancing soft actor-critic using clustered experience replay and multi-dimensional priority sampling," Energy, Elsevier, vol. 319(C).
    4. Liu, Weirong & Yao, Pengfei & Wu, Yue & Duan, Lijun & Li, Heng & Peng, Jun, 2025. "Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system," Applied Energy, Elsevier, vol. 378(PA).
    5. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    6. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    7. Chang, Chengcheng & Zhao, Wanzhong & Wang, Chunyan & Luan, Zhongkai, 2023. "An energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating conditions," Energy, Elsevier, vol. 283(C).
    8. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    9. Chen, Fujun & Wang, Bowen & Ni, Meng & Gong, Zhichao & Jiao, Kui, 2024. "Online energy management strategy for ammonia-hydrogen hybrid electric vehicles harnessing deep reinforcement learning," Energy, Elsevier, vol. 301(C).
    10. Zhang, Dongfang & Sun, Wei & Zou, Yuan & Zhang, Xudong & Zhang, Yiwei, 2024. "An improved soft actor-critic-based energy management strategy of heavy-duty hybrid electric vehicles with dual-engine system," Energy, Elsevier, vol. 308(C).
    11. Liu, Zemin Eitan & Li, Yong & Zhou, Quan & Shuai, Bin & Hua, Min & Xu, Hongming & Xu, Lubing & Tan, Guikun & Li, Yanfei, 2025. "Real-time energy management for HEV combining naturalistic driving data and deep reinforcement learning with high generalization," Applied Energy, Elsevier, vol. 377(PA).
    12. Wang, Jinhai & Du, Changqing & Yan, Fuwu & Hua, Min & Gongye, Xiangyu & Yuan, Quan & Xu, Hongming & Zhou, Quan, 2025. "Bayesian optimization for hyper-parameter tuning of an improved twin delayed deep deterministic policy gradients based energy management strategy for plug-in hybrid electric vehicles," Applied Energy, Elsevier, vol. 381(C).
    13. Li, Jianwei & Liu, Jie & Yang, Qingqing & Wang, Tianci & He, Hongwen & Wang, Hanxiao & Sun, Fengchun, 2025. "Reinforcement learning based energy management for fuel cell hybrid electric vehicles: A comprehensive review on decision process reformulation and strategy implementation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 213(C).
    14. Álvaro Gómez-Barroso & Iban Vicente Makazaga & Ekaitz Zulueta, 2024. "Optimizing Hybrid Electric Vehicle Performance: A Detailed Overview of Energy Management Strategies," Energies, MDPI, vol. 18(1), pages 1-32, December.
    15. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    16. Chen, Xiang & Wang, Xu & Zhao, Wanzhong & Wang, Chunyan & Cheng, Shuo & Luan, Zhongkai, 2025. "Hierarchical deep reinforcement learning based multi-agent game control for energy consumption and traffic efficiency improving of autonomous vehicles," Energy, Elsevier, vol. 323(C).
    17. Kim, Sunwoo & Choi, Yechan & Park, Joungho & Adams, Derrick & Heo, Seongmin & Lee, Jay H., 2024. "Multi-period, multi-timescale stochastic optimization model for simultaneous capacity investment and energy management decisions for hybrid Micro-Grids with green hydrogen production under uncertainty," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    18. Gnann, Till & Stephens, Thomas S. & Lin, Zhenhong & Plötz, Patrick & Liu, Changzheng & Brokate, Jens, 2018. "What drives the market for plug-in electric vehicles? - A review of international PEV market diffusion models," Renewable and Sustainable Energy Reviews, Elsevier, vol. 93(C), pages 158-164.
    19. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    20. Bảo-Huy Nguyễn & João Pedro F. Trovão & Ronan German & Alain Bouscayrol, 2020. "Real-Time Energy Management of Parallel Hybrid Electric Vehicles Using Linear Quadratic Regulation," Energies, MDPI, vol. 13(21), pages 1-19, October.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:288:y:2024:i:c:s0360544223031675. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.