IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v8y2015i7p7243-7260d52695.html
   My bibliography  Save this article

Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle

Author

Listed:
  • Teng Liu

    (Collaborative Innovation Center of Electric Vehicles in Beijing, School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China)

  • Yuan Zou

    (Collaborative Innovation Center of Electric Vehicles in Beijing, School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China)

  • Dexing Liu

    (Collaborative Innovation Center of Electric Vehicles in Beijing, School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China)

  • Fengchun Sun

    (Collaborative Innovation Center of Electric Vehicles in Beijing, School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China)

Abstract

This paper presents a reinforcement learning (RL)–based energy management strategy for a hybrid electric tracked vehicle. A control-oriented model of the powertrain and vehicle dynamics is first established. According to the sample information of the experimental driving schedule, statistical characteristics at various velocities are determined by extracting the transition probability matrix of the power request. Two RL-based algorithms, namely Q -learning and Dyna algorithms, are applied to generate optimal control solutions. The two algorithms are simulated on the same driving schedule, and the simulation results are compared to clarify the merits and demerits of these algorithms. Although the Q -learning algorithm is faster (3 h) than the Dyna algorithm (7 h), its fuel consumption is 1.7% higher than that of the Dyna algorithm. Furthermore, the Dyna algorithm registers approximately the same fuel consumption as the dynamic programming–based global optimal solution. The computational cost of the Dyna algorithm is substantially lower than that of the stochastic dynamic programming.

Suggested Citation

  • Teng Liu & Yuan Zou & Dexing Liu & Fengchun Sun, 2015. "Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle," Energies, MDPI, vol. 8(7), pages 1-18, July.
  • Handle: RePEc:gam:jeners:v:8:y:2015:i:7:p:7243-7260:d:52695
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/8/7/7243/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1996-1073/8/7/7243/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Zou Yuan & Liu Teng & Sun Fengchun & Huei Peng, 2013. "Comparative Study of Dynamic Programming and Pontryagin’s Minimum Principle on Energy Management for a Parallel Hybrid Electric Vehicle," Energies, MDPI, vol. 6(4), pages 1-14, April.
    2. Zhongyue Zou & Jun Xu & Chris Mi & Binggang Cao & Zheng Chen, 2014. "Evaluation of Model Based State of Charge Estimation Methods for Lithium-Ion Batteries," Energies, MDPI, vol. 7(8), pages 1-18, August.
    3. Felipe Jiménez & Wilmar Cabrera-Montiel, 2014. "System for Road Vehicle Energy Optimization Using Real Time Road and Traffic Information," Energies, MDPI, vol. 7(6), pages 1-23, June.
    4. Yuan Zou & Fengchun Sun & Xiaosong Hu & Lino Guzzella & Huei Peng, 2012. "Combined Optimal Sizing and Control for a Hybrid Tracked Vehicle," Energies, MDPI, vol. 5(11), pages 1-14, November.
    5. Abdelsalam Ahmed Abdelsalam & Shumei Cui, 2012. "A Fuzzy Logic Global Power Management Strategy for Hybrid Electric Vehicles Based on a Permanent Magnet Electric Variable Transmission," Energies, MDPI, vol. 5(4), pages 1-24, April.
    6. Bingtuan Gao & Wenhu Zhang & Yi Tang & Mingjin Hu & Mingcheng Zhu & Huiyu Zhan, 2014. "Game-Theoretic Energy Management for Residential Users with Dischargeable Plug-in Electric Vehicles," Energies, MDPI, vol. 7(11), pages 1-20, November.
    7. Tobias Nüesch & Alberto Cerofolini & Giorgio Mancini & Nicolò Cavina & Christopher Onder & Lino Guzzella, 2014. "Equivalent Consumption Minimization Strategy for the Control of Real Driving NOx Emissions of a Diesel Hybrid Electric Vehicle," Energies, MDPI, vol. 7(5), pages 1-31, May.
    8. Tobias Nüesch & Philipp Elbert & Michael Flankl & Christopher Onder & Lino Guzzella, 2014. "Convex Optimization for the Energy Management of Hybrid Electric Vehicles Considering Engine Start and Gearshift Costs," Energies, MDPI, vol. 7(2), pages 1-23, February.
    9. Jingang Guo & Xiaoping Jian & Guangyu Lin, 2014. "Performance Evaluation of an Anti-Lock Braking System for Electric Vehicles with a Fuzzy Sliding Mode Controller," Energies, MDPI, vol. 7(10), pages 1-18, October.
    10. Tri-Vien Vu & Chih-Keng Chen & Chih-Wei Hung, 2014. "A Model Predictive Control Approach for Fuel Economy Improvement of a Series Hydraulic Hybrid Vehicle," Energies, MDPI, vol. 7(11), pages 1-24, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Liu, Teng & Tan, Wenhao & Tang, Xiaolin & Zhang, Jinwei & Xing, Yang & Cao, Dongpu, 2021. "Driving conditions-driven energy management strategies for hybrid electric vehicles: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    2. Bo Hu & Jiaxi Li & Shuang Li & Jie Yang, 2019. "A Hybrid End-to-End Control Strategy Combining Dueling Deep Q-network and PID for Transient Boost Control of a Diesel Engine with Variable Geometry Turbocharger and Cooled EGR," Energies, MDPI, vol. 12(19), pages 1-15, September.
    3. Zhen Zhang & Cheng Ma & Rong Zhu, 2018. "Thermal and Energy Management Based on Bimodal Airflow-Temperature Sensing and Reinforcement Learning," Energies, MDPI, vol. 11(10), pages 1-14, September.
    4. Tran, Dai-Duong & Vafaeipour, Majid & El Baghdadi, Mohamed & Barrero, Ricardo & Van Mierlo, Joeri & Hegazy, Omar, 2020. "Thorough state-of-the-art analysis of electric and hybrid vehicle powertrains: Topologies and integrated energy management strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 119(C).
    5. Du, Guodong & Zou, Yuan & Zhang, Xudong & Guo, Lingxiong & Guo, Ningyuan, 2022. "Energy management for a hybrid electric vehicle based on prioritized deep reinforcement learning framework," Energy, Elsevier, vol. 241(C).
    6. Han, Xuefeng & He, Hongwen & Wu, Jingda & Peng, Jiankun & Li, Yuecheng, 2019. "Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle," Applied Energy, Elsevier, vol. 254(C).
    7. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    8. Huijun Yue & Jinyu Lin & Peng Dong & Zhinan Chen & Xiangyang Xu, 2023. "Configurations and Control Strategies of Hybrid Powertrain Systems," Energies, MDPI, vol. 16(2), pages 1-18, January.
    9. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    10. Xiaobin Ning & Jiazheng Wang & Yuming Yin & Jiarong Shangguan & Nanxin Bao & Ning Li, 2023. "Regenerative Braking Algorithm for Parallel Hydraulic Hybrid Vehicles Based on Fuzzy Q-Learning," Energies, MDPI, vol. 16(4), pages 1-18, February.
    11. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    12. Liu, Teng & Wang, Bo & Yang, Chenglang, 2018. "Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning," Energy, Elsevier, vol. 160(C), pages 544-555.
    13. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    14. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    15. Sun, Wenjing & Zou, Yuan & Zhang, Xudong & Guo, Ningyuan & Zhang, Bin & Du, Guodong, 2022. "High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning," Energy, Elsevier, vol. 258(C).
    16. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    17. Mingliang Bai & Wenjiang Yang & Dongbin Song & Marek Kosuda & Stanislav Szabo & Pavol Lipovsky & Afshar Kasaei, 2020. "Research on Energy Management of Hybrid Unmanned Aerial Vehicles to Improve Energy-Saving and Emission Reduction Performance," IJERPH, MDPI, vol. 17(8), pages 1-24, April.
    18. Wu, Changcheng & Ruan, Jiageng & Cui, Hanghang & Zhang, Bin & Li, Tongyang & Zhang, Kaixuan, 2023. "The application of machine learning based energy management strategy in multi-mode plug-in hybrid electric vehicle, part I: Twin Delayed Deep Deterministic Policy Gradient algorithm design for hybrid ," Energy, Elsevier, vol. 262(PB).
    19. Zou, Yuan & Liu, Teng & Liu, Dexing & Sun, Fengchun, 2016. "Reinforcement learning-based real-time energy management for a hybrid tracked vehicle," Applied Energy, Elsevier, vol. 171(C), pages 372-382.
    20. Jau-Woei Perng & Yi-Horng Lai, 2016. "Robust Longitudinal Speed Control of Hybrid Electric Vehicles with a Two-Degree-of-Freedom Fuzzy Logic Controller," Energies, MDPI, vol. 9(4), pages 1-15, April.
    21. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    22. Hong Huang & Li Zhai & Zeda Wang, 2018. "A Power Coupling System for Electric Tracked Vehicles during High-Speed Steering with Optimization-Based Torque Distribution Control," Energies, MDPI, vol. 11(6), pages 1-17, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Zheng & Xia, Bing & You, Chenwen & Mi, Chunting Chris, 2015. "A novel energy management method for series plug-in hybrid electric vehicles," Applied Energy, Elsevier, vol. 145(C), pages 172-179.
    2. Liu, Teng & Tan, Wenhao & Tang, Xiaolin & Zhang, Jinwei & Xing, Yang & Cao, Dongpu, 2021. "Driving conditions-driven energy management strategies for hybrid electric vehicles: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    3. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    4. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    5. Balerna, Camillo & Lanzetti, Nicolas & Salazar, Mauro & Cerofolini, Alberto & Onder, Christopher, 2020. "Optimal low-level control strategies for a high-performance hybrid electric power unit," Applied Energy, Elsevier, vol. 276(C).
    6. Tobias Nüesch & Philipp Elbert & Michael Flankl & Christopher Onder & Lino Guzzella, 2014. "Convex Optimization for the Energy Management of Hybrid Electric Vehicles Considering Engine Start and Gearshift Costs," Energies, MDPI, vol. 7(2), pages 1-23, February.
    7. Duhr, Pol & Christodoulou, Grigorios & Balerna, Camillo & Salazar, Mauro & Cerofolini, Alberto & Onder, Christopher H., 2021. "Time-optimal gearshift and energy management strategies for a hybrid electric race car," Applied Energy, Elsevier, vol. 282(PA).
    8. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    9. Camillo Balerna & Marc-Philippe Neumann & Nicolò Robuschi & Pol Duhr & Alberto Cerofolini & Vittorio Ravaglioli & Christopher Onder, 2020. "Time-Optimal Low-Level Control and Gearshift Strategies for the Formula 1 Hybrid Electric Powertrain," Energies, MDPI, vol. 14(1), pages 1-30, December.
    10. Yuping Zeng & Yang Cai & Changbao Chu & Guiyue Kou & Wei Gao, 2018. "Integrated Energy and Catalyst Thermal Management for Plug-In Hybrid Electric Vehicles," Energies, MDPI, vol. 11(7), pages 1-29, July.
    11. Zhang, Pei & Yan, Fuwu & Du, Changqing, 2015. "A comprehensive analysis of energy management strategies for hybrid electric vehicles based on bibliometrics," Renewable and Sustainable Energy Reviews, Elsevier, vol. 48(C), pages 88-104.
    12. Caiyang Wei & Theo Hofman & Esin Ilhan Caarls & Rokus van Iperen, 2020. "A Review of the Integrated Design and Control of Electrified Vehicles," Energies, MDPI, vol. 13(20), pages 1, October.
    13. Zou, Yuan & Liu, Teng & Liu, Dexing & Sun, Fengchun, 2016. "Reinforcement learning-based real-time energy management for a hybrid tracked vehicle," Applied Energy, Elsevier, vol. 171(C), pages 372-382.
    14. Wei, Shouyang & Zou, Yuan & Sun, Fengchun & Christopher, Onder, 2017. "A pseudospectral method for solving optimal control problem of a hybrid tracked vehicle," Applied Energy, Elsevier, vol. 194(C), pages 588-595.
    15. Yuping Zeng & Yang Cai & Guiyue Kou & Wei Gao & Datong Qin, 2018. "Energy Management for Plug-In Hybrid Electric Vehicle Based on Adaptive Simplified-ECMS," Sustainability, MDPI, vol. 10(6), pages 1-24, June.
    16. Jixiang Fan & Jiangyan Zhang & Tielong Shen, 2015. "Map-Based Power-Split Strategy Design with Predictive Performance Optimization for Parallel Hybrid Electric Vehicles," Energies, MDPI, vol. 8(9), pages 1-23, September.
    17. Di Guo & Changqing Du & Fuwu Yan, 2016. "Drivability-Related Discrete-Time Model Predictive Control of Mode Transition in Pre-Transmission Parallel Hybrid Powertrains," Energies, MDPI, vol. 9(9), pages 1-31, September.
    18. Johannes Schalk & Harald Aschemann, 2016. "A Causal and Real-Time Capable Power Management Algorithm for Off-Highway Hybrid Propulsion Systems," Energies, MDPI, vol. 10(1), pages 1-17, December.
    19. Felipe Jiménez & Wilmar Cabrera-Montiel, 2014. "System for Road Vehicle Energy Optimization Using Real Time Road and Traffic Information," Energies, MDPI, vol. 7(6), pages 1-23, June.
    20. Shaobo Xie & Xiaosong Hu & Kun Lang & Shanwei Qi & Tong Liu, 2018. "Powering Mode-Integrated Energy Management Strategy for a Plug-In Hybrid Electric Truck with an Automatic Mechanical Transmission Based on Pontryagin’s Minimum Principle," Sustainability, MDPI, vol. 10(10), pages 1-23, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:8:y:2015:i:7:p:7243-7260:d:52695. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.