IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v348y2023ics0306261923008541.html
   My bibliography  Save this article

A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

Author

Listed:
  • Zhang, Shulei
  • Jia, Runda
  • Pan, Hengxin
  • Cao, Yankai

Abstract

With the growing popularity of electric vehicles (EVs), it is a new challenge for the residential microgrid system to conduct charging scheduling to meet the charging demands of EVs while maximizing its profit. In this work, a safe reinforcement learning (RL)-based charging scheduling strategy is proposed to meet this challenge. We construct a complete microgrid system equipped with a large charging station and consider different types of EVs, as well as the vehicle-to-grid (V2G) mode and nonlinear charging characteristics of EVs. Subsequently, the charging scheduling problem is formulated as a constrained Markov decision process (CMDP) due to the various limitations of power and demands. To effectively capture the uncertainties of the supply side and demand side of the microgrid, a model-free RL framework is employed. However, the curse of dimensionality of the action space is inevitable as EVs increase. To solve this problem, a charging and discharging strategy based on a general ladder electricity pricing scheme is designed. Different EVs are divided into different sets according to their states under this strategy, and the agent gives control signals of different sets instead of controlling each EV individually, which effectively reduces the dimension of the action space. Subsequently, a constrained soft actor-critic (CSAC) algorithm is designed to solve the established CMDP, and a safety filter is introduced to ensure safety. In the end, a numerical case is conducted to verify the effectiveness of the proposed method.

Suggested Citation

  • Zhang, Shulei & Jia, Runda & Pan, Hengxin & Cao, Yankai, 2023. "A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid," Applied Energy, Elsevier, vol. 348(C).
    • Handle: RePEc:eee:appene:v:348:y:2023:i:c:s0306261923008541
    DOI: 10.1016/j.apenergy.2023.121490
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923008541
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.121490?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Siobhan Powell & Gustavo Vianna Cezar & Liang Min & Inês M. L. Azevedo & Ram Rajagopal, 2022. "Charging infrastructure access and operation to reduce the grid impacts of deep electric vehicle adoption," Nature Energy, Nature, vol. 7(10), pages 932-945, October.
    2. Yuxin Wen & Peixiao Fan & Jia Hu & Song Ke & Fuzhang Wu & Xu Zhu, 2022. "An Optimal Scheduling Strategy of a Microgrid with V2G Based on Deep Q-Learning," Sustainability, MDPI, vol. 14(16), pages 1-18, August.
    3. Lyu, Yizheng & Gao, Hanbo & Yan, Kun & Liu, Yingjie & Tian, Jinping & Chen, Lyujun & Wan, Mei, 2022. "Carbon peaking strategies for industrial parks: Model development and applications in China," Applied Energy, Elsevier, vol. 322(C).
    4. Dong, Xiangxiang & Wu, Jiang & Xu, Zhanbo & Liu, Kun & Guan, Xiaohong, 2022. "Optimal coordination of hydrogen-based integrated energy systems with combination of hydrogen and water storage," Applied Energy, Elsevier, vol. 308(C).
    5. Athanasios Paraskevas & Dimitrios Aletras & Antonios Chrysopoulos & Antonios Marinopoulos & Dimitrios I. Doukas, 2022. "Optimal Management for EV Charging Stations: A Win–Win Strategy for Different Stakeholders Using Constrained Deep Q-Learning," Energies, MDPI, vol. 15(7), pages 1-24, March.
    6. Lai, Chun Sing & Chen, Dashen & Zhang, Jinning & Zhang, Xin & Xu, Xu & Taylor, Gareth A. & Lai, Loi Lei, 2022. "Profit maximization for large-scale energy storage systems to enable fast EV charging infrastructure in distribution networks," Energy, Elsevier, vol. 259(C).
    7. Lee, Sangyoon & Choi, Dae-Hyun, 2021. "Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach," Applied Energy, Elsevier, vol. 304(C).
    8. Zhao, Shihao & Li, Kang & Yang, Zhile & Xu, Xinzhi & Zhang, Ning, 2022. "A new power system active rescheduling method considering the dispatchable plug-in electric vehicles and intermittent renewable energies," Applied Energy, Elsevier, vol. 314(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Athanasios Ioannis Arvanitidis & Vivek Agarwal & Miltiadis Alamaniotis, 2023. "Nuclear-Driven Integrated Energy Systems: A State-of-the-Art Review," Energies, MDPI, vol. 16(11), pages 1-23, May.
    2. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    3. Ma, Huan & Sun, Qinghan & Chen, Qun & Zhao, Tian & He, Kelun, 2023. "Exergy-based flexibility cost indicator and spatio-temporal coordination principle of distributed multi-energy systems," Energy, Elsevier, vol. 267(C).
    4. Yin, Rumeng & He, Jiang, 2023. "Design of a photovoltaic electric bike battery-sharing system in public transit stations," Applied Energy, Elsevier, vol. 332(C).
    5. Alya AlHammadi & Nasser Al-Saif & Ameena Saad Al-Sumaiti & Mousa Marzband & Tareefa Alsumaiti & Ehsan Heydarian-Forushani, 2022. "Techno-Economic Analysis of Hybrid Renewable Energy Systems Designed for Electric Vehicle Charging: A Case Study from the United Arab Emirates," Energies, MDPI, vol. 15(18), pages 1-20, September.
    6. Subhojit Dawn & Gummadi Srinivasa Rao & M. L. N. Vital & K. Dhananjay Rao & Faisal Alsaif & Mohammed H. Alsharif, 2023. "Profit Extension of a Wind-Integrated Competitive Power System by Vehicle-to-Grid Integration and UPFC Placement," Energies, MDPI, vol. 16(18), pages 1-24, September.
    7. Xingyun Yan & Lingyu Wang & Mingzhu Fang & Jie Hu, 2022. "How Can Industrial Parks Achieve Carbon Neutrality? Literature Review and Research Prospect Based on the CiteSpace Knowledge Map," Sustainability, MDPI, vol. 15(1), pages 1-29, December.
    8. Zhao, Zhonghao & Lee, Carman K.M. & Huo, Jiage, 2023. "EV charging station deployment on coupled transportation and power distribution networks via reinforcement learning," Energy, Elsevier, vol. 267(C).
    9. Alejandro Sanz & Peter Meyer, 2024. "Electrifying the Last-Mile Logistics (LML) in Intensive B2B Operations—An European Perspective on Integrating Innovative Platforms," Logistics, MDPI, vol. 8(2), pages 1-39, April.
    10. Sohani, Ali & Cornaro, Cristina & Shahverdian, Mohammad Hassan & Moser, David & Pierro, Marco & Olabi, Abdul Ghani & Karimi, Nader & Nižetić, Sandro & Li, Larry K.B. & Doranehgard, Mohammad Hossein, 2023. "Techno-economic evaluation of a hybrid photovoltaic system with hot/cold water storage for poly-generation in a residential building," Applied Energy, Elsevier, vol. 331(C).
    11. Cui, Li & Wang, Qingyuan & Qu, Hongquan & Wang, Mingshen & Wu, Yile & Ge, Le, 2023. "Dynamic pricing for fast charging stations with deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    12. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    13. Ibrahim Alsaidan & Mohd Bilal & Muhannad Alaraj & Mohammad Rizwan & Fahad M. Almasoudi, 2023. "A Novel EA-Based Techno–Economic Analysis of Charging System for Electric Vehicles: A Case Study of Qassim Region, Saudi Arabia," Mathematics, MDPI, vol. 11(9), pages 1-31, April.
    14. George-Williams, H. & Wade, N. & Carpenter, R.N., 2022. "A probabilistic framework for the techno-economic assessment of smart energy hubs for electric vehicle charging," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    15. Chuming Pang & Yongkui Shi & Yang Liu, 2023. "Research on Sustainable Development of Mining Goaf Management Based on Economic Models," Sustainability, MDPI, vol. 15(20), pages 1-21, October.
    16. Chen, Minghao & Sun, Yi & Xie, Zhiyuan & Lin, Nvgui & Wu, Peng, 2023. "An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning," Energy, Elsevier, vol. 284(C).
    17. Li, Xiaohui & Wang, Zhenpo & Zhang, Lei & Sun, Fengchun & Cui, Dingsong & Hecht, Christopher & Figgener, Jan & Sauer, Dirk Uwe, 2023. "Electric vehicle behavior modeling and applications in vehicle-grid integration: An overview," Energy, Elsevier, vol. 268(C).
    18. Bin Xiao & Zhenxin Gao & Huaiwu Peng & Kang Chen & Yang Li & Kun Liu, 2023. "Robust Optimization of Large-Scale Wind–Solar Storage Renewable Energy Systems Considering Hybrid Storage Multi-Energy Synergy," Sustainability, MDPI, vol. 16(1), pages 1-21, December.
    19. Xin Zhou & Xin Meng & Zhenyu Li, 2024. "ANN-LSTM-A Water Consumption Prediction Based on Attention Mechanism Enhancement," Energies, MDPI, vol. 17(5), pages 1-16, February.
    20. Fengyuan Yan & Xiaolong Han & Qianwei Cheng & Yamin Yan & Qi Liao & Yongtu Liang, 2022. "Scenario-Based Comparative Analysis for Coupling Electricity and Hydrogen Storage in Clean Oilfield Energy Supply System," Energies, MDPI, vol. 15(6), pages 1-28, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:348:y:2023:i:c:s0306261923008541. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.