IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v313y2022ics0306261922002379.html
   My bibliography  Save this article

Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach

Author

Listed:
  • Qiu, Dawei
  • Wang, Yi
  • Sun, Mingyang
  • Strbac, Goran

Abstract

In order to achieve the target of carbon peak and carbon neutrality, electric vehicles (EVs) have increasingly received a prominent interest to electrify the transportation sector due to their advantages of mobility and flexibility on handling complicated transportation and power networks. However, it is still challenging to realize the significant potential of EVs towards an emerging low-carbon transition. Previous works have focused on vehicle-to-grid (V2G) technology that allows for an increased utilization of EVs to make arbitrage by the temporal differentials of electricity prices. Nevertheless, the economic potential of EVs flexibility may not be fully exploited lacking an appropriate business model. This paper addresses this challenge by developing a coupled power-transportation network for cooperative EVs to optimize the provision of multiple inter-dependent services, including charging service, demand management service, carbon intensity service, and balancing service. In order to unlock this value, the EVs operation problem has already been tackled using model-based optimization approaches, which may raise privacy issues since the requirement for global information and also can be time consuming due to the high variability of transportation and power networks. In this paper, we propose a model-free hierarchical and hybrid multi-agent reinforcement learning method to learn the routing and scheduling decisions of EVs in a coupled power-transportation network with the objective of optimizing multi-service provisions. To this end, EVs do not reply on any knowledge of the simulated environment and are capable of handling system uncertainties via the learning process. Extensive case studies based on a 15-bus radial power distribution network and a 9-node 12-edge transportation network are developed to show that the proposed method outperforms the conventional learning algorithms in terms of policy quality and convergence speed. Finally, the generalizability and scalability are also investigated for different environment circumstances and EV numbers.

Suggested Citation

  • Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    • Handle: RePEc:eee:appene:v:313:y:2022:i:c:s0306261922002379
    DOI: 10.1016/j.apenergy.2022.118790
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922002379
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.118790?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhou, Quan & Li, Yanfei & Zhao, Dezong & Li, Ji & Williams, Huw & Xu, Hongming & Yan, Fuwu, 2022. "Transferable representation modelling for real-time energy management of the plug-in hybrid vehicle based on k-fold fuzzy learning and Gaussian process regression," Applied Energy, Elsevier, vol. 305(C).
    2. Rezkalla, Michel & Zecchino, Antonio & Martinenas, Sergejus & Prostejovsky, Alexander M. & Marinelli, Mattia, 2018. "Comparison between synthetic inertia and fast frequency containment control based on single phase EVs in a microgrid," Applied Energy, Elsevier, vol. 210(C), pages 764-775.
    3. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    4. DeForest, Nicholas & MacDonald, Jason S. & Black, Douglas R., 2018. "Day ahead optimization of an electric vehicle fleet providing ancillary services in the Los Angeles Air Force Base vehicle-to-grid demonstration," Applied Energy, Elsevier, vol. 210(C), pages 987-1001.
    5. Zhou, Yue & Wu, Jianzhong & Song, Guanyu & Long, Chao, 2020. "Framework design and optimal bidding strategy for ancillary service provision from a peer-to-peer energy trading community," Applied Energy, Elsevier, vol. 278(C).
    6. Dixon, James & Bukhsh, Waqquas & Edmunds, Calum & Bell, Keith, 2020. "Scheduling electric vehicle charging to minimise carbon emissions and wind curtailment," Renewable Energy, Elsevier, vol. 161(C), pages 1072-1091.
    7. Daryabari, Mohamad K. & Keypour, Reza & Golmohamadi, Hessam, 2020. "Stochastic energy management of responsive plug-in electric vehicles characterizing parking lot aggregators," Applied Energy, Elsevier, vol. 279(C).
    8. Gan, Wei & Yan, Mingyu & Yao, Wei & Wen, Jinyu, 2021. "Peer to peer transactive energy for multiple energy hub with the penetration of high-level renewable energy," Applied Energy, Elsevier, vol. 295(C).
    9. Shuai, Bin & Zhou, Quan & Li, Ji & He, Yinglong & Li, Ziyang & Williams, Huw & Xu, Hongming & Shuai, Shijin, 2020. "Heuristic action execution for energy efficient charge-sustaining control of connected hybrid vehicles with model-free double Q-learning," Applied Energy, Elsevier, vol. 267(C).
    10. Zhou, Quan & Li, Ji & Shuai, Bin & Williams, Huw & He, Yinglong & Li, Ziyang & Xu, Hongming & Yan, Fuwu, 2019. "Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle," Applied Energy, Elsevier, vol. 255(C).
    11. Das, Ridoy & Wang, Yue & Putrus, Ghanim & Kotter, Richard & Marzband, Mousa & Herteleer, Bert & Warmerdam, Jos, 2020. "Multi-objective techno-economic-environmental optimisation of electric vehicle for energy services," Applied Energy, Elsevier, vol. 257(C).
    12. Wei, Hongqian & Zhang, Youtong & Wang, Yongzhen & Hua, Weiqi & Jing, Rui & Zhou, Yue, 2022. "Planning integrated energy systems coupling V2G as a flexible storage," Energy, Elsevier, vol. 239(PB).
    13. Hui, Hongxun & Ding, Yi & Shi, Qingxin & Li, Fangxing & Song, Yonghua & Yan, Jinyue, 2020. "5G network-based Internet of Things for demand response in smart grid: A survey on application potential," Applied Energy, Elsevier, vol. 257(C).
    14. Liao, Zitong & Taiebat, Morteza & Xu, Ming, 2021. "Shared autonomous electric vehicle fleets with vehicle-to-grid capability: Economic viability and environmental co-benefits," Applied Energy, Elsevier, vol. 302(C).
    15. Mishra, Sakshi & Anderson, Kate & Miller, Brian & Boyer, Kyle & Warren, Adam, 2020. "Microgrid resilience: A holistic approach for assessing threats, identifying vulnerabilities, and designing corresponding mitigation strategies," Applied Energy, Elsevier, vol. 264(C).
    16. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    17. Shang, Wen-Long & Chen, Jinyu & Bi, Huibo & Sui, Yi & Chen, Yanyan & Yu, Haitao, 2021. "Impacts of COVID-19 pandemic on user behaviors and environmental benefits of bike sharing: A big-data analysis," Applied Energy, Elsevier, vol. 285(C).
    18. Wen-Long Shang & Yanyan Chen & Huibo Bi & Haoran Zhang & Changxi Ma & Washington Y. Ochieng, 2020. "Statistical Characteristics and Community Analysis of Urban Road Networks," Complexity, Hindawi, vol. 2020, pages 1-21, September.
    19. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    20. Bi, Huibo & Shang, Wen-Long & Chen, Yanyan & Wang, Kezhi & Yu, Qing & Sui, Yi, 2021. "GIS aided sustainable urban road management with a unifying queueing and neural network model," Applied Energy, Elsevier, vol. 291(C).
    21. Noori, Mehdi & Zhao, Yang & Onat, Nuri C. & Gardner, Stephanie & Tatari, Omer, 2016. "Light-duty electric vehicles to improve the integrity of the electricity grid through Vehicle-to-Grid technology: Analysis of regional net revenue and emissions savings," Applied Energy, Elsevier, vol. 168(C), pages 146-158.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    2. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    3. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    4. Anna Auza & Ehsan Asadi & Behrang Chenari & Manuel Gameiro da Silva, 2023. "A Systematic Review of Uncertainty Handling Approaches for Electric Grids Considering Electrical Vehicles," Energies, MDPI, vol. 16(13), pages 1-25, June.
    5. Song, Wenchao & Lu, Chao & Lin, Junjie & Fang, Chen & Liu, Shu, 2023. "A low-quality PMU data identification method with dynamic criteria based on spatial–temporal correlations and random matrices," Applied Energy, Elsevier, vol. 343(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Yu, Qing & Li, Weifeng & Zhang, Haoran & Chen, Jinyu, 2022. "GPS data in taxi-sharing system: Analysis of potential demand and assessment of fuel consumption based on routing probability model," Applied Energy, Elsevier, vol. 314(C).
    4. Zhou, Junfeng & Zhang, Yanhui & Zhang, Yubo & Shang, Wen-Long & Yang, Zhile & Feng, Wei, 2022. "Parameters identification of photovoltaic models using a differential evolution algorithm based on elite and obsolete dynamic learning," Applied Energy, Elsevier, vol. 314(C).
    5. Azim, M. Imran & Tushar, Wayes & Saha, Tapan K. & Yuen, Chau & Smith, David, 2022. "Peer-to-peer kilowatt and negawatt trading: A review of challenges and recent advances in distribution networks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 169(C).
    6. Hua, Min & Zhang, Cetengfei & Zhang, Fanggang & Li, Zhi & Yu, Xiaoli & Xu, Hongming & Zhou, Quan, 2023. "Energy management of multi-mode plug-in hybrid electric vehicle using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 348(C).
    7. Qiu, Dawei & Wang, Yi & Zhang, Tingqi & Sun, Mingyang & Strbac, Goran, 2023. "Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience," Applied Energy, Elsevier, vol. 336(C).
    8. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    9. Dai, Rongjian & Ding, Chuan & Gao, Jian & Wu, Xinkai & Yu, Bin, 2022. "Optimization and evaluation for autonomous taxi ride-sharing schedule and depot location from the perspective of energy consumption," Applied Energy, Elsevier, vol. 308(C).
    10. Zhou, Quan & Li, Yanfei & Zhao, Dezong & Li, Ji & Williams, Huw & Xu, Hongming & Yan, Fuwu, 2022. "Transferable representation modelling for real-time energy management of the plug-in hybrid vehicle based on k-fold fuzzy learning and Gaussian process regression," Applied Energy, Elsevier, vol. 305(C).
    11. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    12. Yao, Zhaosheng & Wang, Zhiyuan & Ran, Lun, 2023. "Smart charging and discharging of electric vehicles based on multi-objective robust optimization in smart cities," Applied Energy, Elsevier, vol. 343(C).
    13. Wenjing Wang & Yanyan Chen & Haodong Sun & Yusen Chen, 2021. "Multiple Binary Classification Model of Trip Chain Based on the Fusion of Internet Location Data and Transport Data," Sustainability, MDPI, vol. 13(21), pages 1-15, November.
    14. Shafqat Jawad & Junyong Liu, 2020. "Electrical Vehicle Charging Services Planning and Operation with Interdependent Power Networks and Transportation Networks: A Review of the Current Scenario and Future Trends," Energies, MDPI, vol. 13(13), pages 1-24, July.
    15. Zhou, Yuekuan & Lund, Peter D., 2023. "Peer-to-peer energy sharing and trading of renewable energy in smart communities ─ trading pricing models, decision-making and agent-based collaboration," Renewable Energy, Elsevier, vol. 207(C), pages 177-193.
    16. Luiz Almeida & Ana Soares & Pedro Moura, 2023. "A Systematic Review of Optimization Approaches for the Integration of Electric Vehicles in Public Buildings," Energies, MDPI, vol. 16(13), pages 1-26, June.
    17. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    18. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    19. Zhao, Shihao & Li, Kang & Yang, Zhile & Xu, Xinzhi & Zhang, Ning, 2022. "A new power system active rescheduling method considering the dispatchable plug-in electric vehicles and intermittent renewable energies," Applied Energy, Elsevier, vol. 314(C).
    20. Qiao, Dongdong & Wei, Xuezhe & Fan, Wenjun & Jiang, Bo & Lai, Xin & Zheng, Yuejiu & Tang, Xiaolin & Dai, Haifeng, 2022. "Toward safe carbon–neutral transportation: Battery internal short circuit diagnosis based on cloud data for electric vehicles," Applied Energy, Elsevier, vol. 317(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:313:y:2022:i:c:s0306261922002379. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.