IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v211y2020ics0360544220320387.html
   My bibliography  Save this article

Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning

Author

Listed:
  • Chen, Zheng
  • Hu, Hengjie
  • Wu, Yitao
  • Zhang, Yuanjian
  • Li, Guang
  • Liu, Yonggang

Abstract

In this paper, a stochastic model predictive control (MPC) method based on reinforcement learning is proposed for energy management of plug-in hybrid electric vehicles (PHEVs). Firstly, the power transfer of each component in a power-split PHEV is described in detail. Then an effective and convergent reinforcement learning controller is trained by the Q-learning algorithm according to the driving power distribution under multiple driving cycles. By constructing a multi-step Markov velocity prediction model, the reinforcement learning controller is embedded into the stochastic MPC controller to determine the optimal battery power in predicted time domain. Numerical simulation results verify that the proposed method achieves superior fuel economy that is close to that by stochastic dynamic programming method. In addition, the effective state of charge tracking in terms of different reference trajectories highlight that the proposed method is effective for online application requiring a fast calculation speed.

Suggested Citation

  • Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    • Handle: RePEc:eee:energy:v:211:y:2020:i:c:s0360544220320387
    DOI: 10.1016/j.energy.2020.118931
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544220320387
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2020.118931?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hou, Cong & Ouyang, Minggao & Xu, Liangfei & Wang, Hewu, 2014. "Approximate Pontryagin’s minimum principle applied to the energy management of plug-in hybrid electric vehicles," Applied Energy, Elsevier, vol. 115(C), pages 174-189.
    2. Hou, Jun & Song, Ziyou, 2020. "A hierarchical energy management strategy for hybrid energy storage via vehicle-to-cloud connectivity," Applied Energy, Elsevier, vol. 257(C).
    3. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    4. Shu, Xing & Li, Guang & Shen, Jiangwei & Lei, Zhenzhen & Chen, Zheng & Liu, Yonggang, 2020. "An adaptive multi-state estimation algorithm for lithium-ion batteries incorporating temperature compensation," Energy, Elsevier, vol. 207(C).
    5. Wieczorek, Maciej & Lewandowski, Mirosław, 2017. "A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm," Applied Energy, Elsevier, vol. 192(C), pages 222-233.
    6. M. Sabri, M.F. & Danapalasingam, K.A. & Rahmat, M.F., 2016. "A review on hybrid electric vehicles architecture and energy management strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 53(C), pages 1433-1442.
    7. Peng, Jiankun & He, Hongwen & Xiong, Rui, 2017. "Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming," Applied Energy, Elsevier, vol. 185(P2), pages 1633-1643.
    8. Zhou, Quan & Li, Ji & Shuai, Bin & Williams, Huw & He, Yinglong & Li, Ziyang & Xu, Hongming & Yan, Fuwu, 2019. "Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle," Applied Energy, Elsevier, vol. 255(C).
    9. Feiyan Qin & Guoqing Xu & Yue Hu & Kun Xu & Weimin Li, 2017. "Stochastic Optimal Control of Parallel Hybrid Electric Vehicles," Energies, MDPI, vol. 10(2), pages 1-16, February.
    10. Wang, Yujie & Sun, Zhendong & Li, Xiyun & Yang, Xiaoyu & Chen, Zonghai, 2019. "A comparative study of power allocation strategies used in fuel cell and ultracapacitor hybrid systems," Energy, Elsevier, vol. 189(C).
    11. Wang, Yujie & Sun, Zhendong & Chen, Zonghai, 2019. "Energy management strategy for battery/supercapacitor/fuel cell hybrid source vehicles based on finite state machine," Applied Energy, Elsevier, vol. 254(C).
    12. Li, Gaopeng & Zhang, Jieli & He, Hongwen, 2017. "Battery SOC constraint comparison for predictive energy management of plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 194(C), pages 578-587.
    13. Du, Jiuyu & Chen, Jingfu & Song, Ziyou & Gao, Mingming & Ouyang, Minggao, 2017. "Design method of a power management strategy for variable battery capacities range-extended electric vehicles to improve energy efficiency and cost-effectiveness," Energy, Elsevier, vol. 121(C), pages 32-42.
    14. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    15. Chen, Zheng & Xia, Bing & You, Chenwen & Mi, Chunting Chris, 2015. "A novel energy management method for series plug-in hybrid electric vehicles," Applied Energy, Elsevier, vol. 145(C), pages 172-179.
    16. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    17. Xie, Shanshan & He, Hongwen & Peng, Jiankun, 2017. "An energy management strategy based on stochastic model predictive control for plug-in hybrid electric buses," Applied Energy, Elsevier, vol. 196(C), pages 279-288.
    18. Hou, Jun & Sun, Jing & Hofmann, Heath, 2018. "Adaptive model predictive control with propulsion load estimation and prediction for all-electric ship energy management," Energy, Elsevier, vol. 150(C), pages 877-889.
    19. Teng Liu & Yuan Zou & Dexing Liu & Fengchun Sun, 2015. "Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle," Energies, MDPI, vol. 8(7), pages 1-18, July.
    20. Du, Yongchang & Zhao, Yue & Wang, Qinpu & Zhang, Yuanbo & Xia, Huaicheng, 2016. "Trip-oriented stochastic optimal energy management strategy for plug-in hybrid electric bus," Energy, Elsevier, vol. 115(P1), pages 1259-1271.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Md. Sazal Miah & Molla Shahadat Hossain Lipu & Sheikh Tanzim Meraj & Kamrul Hasan & Shaheer Ansari & Taskin Jamal & Hasan Masrur & Rajvikram Madurai Elavarasan & Aini Hussain, 2021. "Optimized Energy Management Schemes for Electric Vehicle Applications: A Bibliometric Analysis towards Future Trends," Sustainability, MDPI, vol. 13(22), pages 1-38, November.
    2. Sun, Xilei & Fu, Jianqin & Yang, Huiyong & Xie, Mingke & Liu, Jingping, 2023. "An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control," Energy, Elsevier, vol. 269(C).
    3. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    4. Biswas, Dhrupad & Ghosh, Susenjit & Sengupta, Somnath & Mukhopadhyay, Siddhartha, 2022. "Energy Management of a Parallel Hybrid Electric Vehicle using Model Predictive Static Programming," Energy, Elsevier, vol. 250(C).
    5. Khalil Bachiri & Ali Yahyaouy & Hamid Gualous & Maria Malek & Younes Bennani & Philippe Makany & Nicoleta Rogovschi, 2023. "Multi-Agent DDPG Based Electric Vehicles Charging Station Recommendation," Energies, MDPI, vol. 16(16), pages 1-17, August.
    6. Wei, Changyin & Chen, Yong & Li, Xiaoyu & Lin, Xiaozhe, 2022. "Integrating intelligent driving pattern recognition with adaptive energy management strategy for extender range electric logistics vehicle," Energy, Elsevier, vol. 247(C).
    7. Yang, Dongpo & Liu, Tong & Song, Dafeng & Zhang, Xuanming & Zeng, Xiaohua, 2023. "A real time multi-objective optimization Guided-MPC strategy for power-split hybrid electric bus based on velocity prediction," Energy, Elsevier, vol. 276(C).
    8. Yaqian Wang & Xiaohong Jiao, 2022. "Dual Heuristic Dynamic Programming Based Energy Management Control for Hybrid Electric Vehicles," Energies, MDPI, vol. 15(9), pages 1-19, April.
    9. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    10. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    11. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    12. Tang, Wenbin & Wang, Yaqian & Jiao, Xiaohong & Ren, Lina, 2023. "Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios," Energy, Elsevier, vol. 265(C).
    13. Lin, Xinyou & Wu, Jiayun & Wei, Yimin, 2021. "An ensemble learning velocity prediction-based energy management strategy for a plug-in hybrid electric vehicle considering driving pattern adaptive reference SOC," Energy, Elsevier, vol. 234(C).
    14. Cui, Wei & Cui, Naxin & Li, Tao & Cui, Zhongrui & Du, Yi & Zhang, Chenghui, 2022. "An efficient multi-objective hierarchical energy management strategy for plug-in hybrid electric vehicle in connected scenario," Energy, Elsevier, vol. 257(C).
    15. Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    16. Lin Li & Serdar Coskun & Jiaze Wang & Youming Fan & Fengqi Zhang & Reza Langari, 2021. "Velocity Prediction Based on Vehicle Lateral Risk Assessment and Traffic Flow: A Brief Review and Application Examples," Energies, MDPI, vol. 14(12), pages 1-30, June.
    17. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    18. Kong, Yan & Xu, Nan & Liu, Qiao & Sui, Yan & Yue, Fenglai, 2023. "A data-driven energy management method for parallel PHEVs based on action dependent heuristic dynamic programming (ADHDP) model," Energy, Elsevier, vol. 265(C).
    19. Chen, Zheng & Wu, Simin & Shen, Shiquan & Liu, Yonggang & Guo, Fengxiang & Zhang, Yuanjian, 2023. "Co-optimization of velocity planning and energy management for autonomous plug-in hybrid electric vehicles in urban driving scenarios," Energy, Elsevier, vol. 263(PF).
    20. Vo-Van Thanh & Wencong Su & Bin Wang, 2022. "Optimal DC Microgrid Operation with Model Predictive Control-Based Voltage-Dependent Demand Response and Optimal Battery Dispatch," Energies, MDPI, vol. 15(6), pages 1-19, March.
    21. Zhang, Yuanjian & Liu, Yonggang & Huang, Yanjun & Chen, Zheng & Li, Guang & Hao, Wanming & Cunningham, Geoff & Early, Juliana, 2021. "An optimal control strategy design for plug-in hybrid electric vehicles based on internet of vehicles," Energy, Elsevier, vol. 228(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    4. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    5. Zhuang, Weichao & Li (Eben), Shengbo & Zhang, Xiaowu & Kum, Dongsuk & Song, Ziyou & Yin, Guodong & Ju, Fei, 2020. "A survey of powertrain configuration studies on hybrid electric vehicles," Applied Energy, Elsevier, vol. 262(C).
    6. Zhang, Hailong & Peng, Jiankun & Tan, Huachun & Dong, Hanxuan & Ding, Fan & Ran, Bin, 2020. "Tackling SOC long-term dynamic for energy management of hybrid electric buses via adaptive policy optimization," Applied Energy, Elsevier, vol. 269(C).
    7. Liu, Teng & Wang, Bo & Yang, Chenglang, 2018. "Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning," Energy, Elsevier, vol. 160(C), pages 544-555.
    8. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    9. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    10. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    11. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    12. Wang, Yue & Zeng, Xiaohua & Song, Dafeng & Yang, Nannan, 2019. "Optimal rule design methodology for energy management strategy of a power-split hybrid electric bus," Energy, Elsevier, vol. 185(C), pages 1086-1099.
    13. Hou, Shengyan & Yin, Hai & Xu, Fuguo & Benjamín, Pla & Gao, Jinwu & Chen, Hong, 2023. "Multihorizon predictive energy optimization and lifetime management for connected fuel cell electric vehicles," Energy, Elsevier, vol. 266(C).
    14. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    15. Kong, Yan & Xu, Nan & Zhang, Yuanjian & Sui, Yan & Ju, Hao & Liu, Heng & Xu, Zhe, 2021. "Acquisition of full-factor trip information for global optimization energy management in multi-energy source vehicles and the measure of the amount of information to be transmitted," Energy, Elsevier, vol. 236(C).
    16. Xiao, B. & Ruan, J. & Yang, W. & Walker, P.D. & Zhang, N., 2021. "A review of pivotal energy management strategies for extended range electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    17. Ye Yang & Youtong Zhang & Jingyi Tian & Si Zhang, 2018. "Research on a Plug-In Hybrid Electric Bus Energy Management Strategy Considering Drivability," Energies, MDPI, vol. 11(8), pages 1-22, August.
    18. Xie, Shaobo & Hu, Xiaosong & Xin, Zongke & Brighton, James, 2019. "Pontryagin’s Minimum Principle based model predictive control of energy management for a plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 236(C), pages 893-905.
    19. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    20. Hu, Dong & Xie, Hui & Song, Kang & Zhang, Yuanyuan & Yan, Long, 2023. "An apprenticeship-reinforcement learning scheme based on expert demonstrations for energy management strategy of hybrid electric vehicles," Applied Energy, Elsevier, vol. 342(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:211:y:2020:i:c:s0360544220320387. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.