IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v339y2023ics0306261923002660.html
   My bibliography  Save this article

Novel Data-Driven decentralized coordination model for electric vehicle aggregator and energy hub entities in multi-energy system using an improved multi-agent DRL approach

Author

Listed:
  • Zhang, Bin
  • Hu, Weihao
  • Cao, Di
  • Ghias, Amer M.Y.M.
  • Chen, Zhe

Abstract

Energy hub (EH) is an independent entity that benefits to the efficiency, flexibility, and reliability of integrated energy systems (IESs). On the other hand, the rapid emerging of electric vehicles (EVs) drives the EV aggregator (EVAGG) as another independent entity to facilitate the electricity exchange with the grid. However, due to privacy consideration for different owners, it is challenging to investigate the optimal coordinated strategies for such interconnected entities only by exchanging the information of electrical energy. Besides, the existence of parameter uncertainties (load demands, EVs’ charging behaviors, wind power and photovoltaic generation), continuous decision space, dynamic energy flows, and non-convex multi-objective function is difficult to solve. To this end, this paper proposes a novel model-free multi-agent deep reinforcement learning (MADRL) -based decentralized coordination model to minimize the energy costs of EH entities and maximize profits of EVAGGs. First, a long short-term memory (LSTM) module is used to capture the future trend of uncertainties. Then, the coordination problem is formulated as Markov games and solved by the attention enabled MADRL algorithm, where the EH or EVAGG entity is modeled as an adaptive agent. An attention mechanism makes each agent only focus on state information related to the reward. The proposed MADRL adopts the forms of offline centralized training to learn the optimal coordinated control strategy, and decentralized execution to enable agents’ online decisions to only require local measurements. A safety network is employed to cope with equality constraints (demand–supply balance). Simulation results illustrate that the proposed method achieves similar results compared to the traditional model-based method with perfect knowledge of system models, and the computation performance is at least two orders of magnitudes shorter than the traditional method. The testing results of the proposed method are better than those of the Concurrent and other MADRL method, with 10.79%/3.06% lower energy cost and 17.11%/6.82% higher profits of aggregator. Besides, the electric equality constraint of the proposed method is only 0.25 MW averaged per day, which is a small and acceptable violation.

Suggested Citation

  • Zhang, Bin & Hu, Weihao & Cao, Di & Ghias, Amer M.Y.M. & Chen, Zhe, 2023. "Novel Data-Driven decentralized coordination model for electric vehicle aggregator and energy hub entities in multi-energy system using an improved multi-agent DRL approach," Applied Energy, Elsevier, vol. 339(C).
  • Handle: RePEc:eee:appene:v:339:y:2023:i:c:s0306261923002660
    DOI: 10.1016/j.apenergy.2023.120902
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923002660
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2023.120902?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yang, Wenjun & Guo, Jia & Vartosh, Aris, 2022. "Optimal economic-emission planning of multi-energy systems integrated electric vehicles with modified group search optimization," Applied Energy, Elsevier, vol. 311(C).
    2. Seyfi, Mohammad & Mehdinejad, Mehdi & Mohammadi-Ivatloo, Behnam & Shayanfar, Heidarali, 2022. "Deep learning-based scheduling of virtual energy hubs with plug-in hybrid compressed natural gas-electric vehicles," Applied Energy, Elsevier, vol. 321(C).
    3. Wang, Yi & Gan, Dahua & Sun, Mingyang & Zhang, Ning & Lu, Zongxiang & Kang, Chongqing, 2019. "Probabilistic individual load forecasting using pinball loss guided LSTM," Applied Energy, Elsevier, vol. 235(C), pages 10-20.
    4. Najafi, Arsalan & Pourakbari-Kasmaei, Mahdi & Jasinski, Michal & Lehtonen, Matti & Leonowicz, Zbigniew, 2021. "A hybrid decentralized stochastic-robust model for optimal coordination of electric vehicle aggregator and energy hub entities," Applied Energy, Elsevier, vol. 304(C).
    5. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    6. Xie, Shiwei & Hu, Zhijian & Wang, Jueying & Chen, Yuwei, 2020. "The optimal planning of smart multi-energy systems incorporating transportation, natural gas and active distribution networks," Applied Energy, Elsevier, vol. 269(C).
    7. Jordehi, A. Rezaee & Javadi, Mohammad Sadegh & Catalão, João P.S., 2021. "Day-ahead scheduling of energy hubs with parking lots for electric vehicles considering uncertainties," Energy, Elsevier, vol. 229(C).
    8. Ata, Mustafa & Erenoğlu, Ayşe Kübra & Şengör, İbrahim & Erdinç, Ozan & Taşcıkaraoğlu, Akın & Catalão, João P.S., 2019. "Optimal operation of a multi-energy system considering renewable energy sources stochasticity and impacts of electric vehicles," Energy, Elsevier, vol. 186(C).
    9. Rishee K. Jain & Junjie Qin & Ram Rajagopal, 2017. "Data-driven planning of distributed energy resources amidst socio-technical complexities," Nature Energy, Nature, vol. 2(8), pages 1-11, August.
    10. Zhu, Dafeng & Yang, Bo & Liu, Yuxiang & Wang, Zhaojian & Ma, Kai & Guan, Xinping, 2022. "Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park," Applied Energy, Elsevier, vol. 311(C).
    11. Liu, Haizhou & Shen, Xinwei & Guo, Qinglai & Sun, Hongbin, 2021. "A data-driven approach towards fast economic dispatch in electricity–gas coupled systems based on artificial neural network," Applied Energy, Elsevier, vol. 286(C).
    12. Mittelviefhaus, Moritz & Pareschi, Giacomo & Allan, James & Georges, Gil & Boulouchos, Konstantinos, 2021. "Optimal investment and scheduling of residential multi-energy systems including electric mobility: A cost-effective approach to climate change mitigation," Applied Energy, Elsevier, vol. 301(C).
    13. Jing, Rui & Xie, Mei Na & Wang, Feng Xiang & Chen, Long Xiang, 2020. "Fair P2P energy trading between residential and commercial multi-energy systems enabling integrated demand-side management," Applied Energy, Elsevier, vol. 262(C).
    14. Ashfaq, Asad & Ianakiev, Anton, 2018. "Cost-minimised design of a highly renewable heating network for fossil-free future," Energy, Elsevier, vol. 152(C), pages 613-626.
    15. Klemm, Christian & Vennemann, Peter, 2021. "Modeling and optimization of multi-energy systems in mixed-use districts: A review of existing methods and approaches," Renewable and Sustainable Energy Reviews, Elsevier, vol. 135(C).
    16. Zhang, Suhan & Gu, Wei & Lu, Hai & Qiu, Haifeng & Lu, Shuai & Wang, Dada & Liang, Junyu & Li, Wenyun, 2021. "Superposition-principle based decoupling method for energy flow calculation in district heating networks," Applied Energy, Elsevier, vol. 295(C).
    17. Wang, Yi & Zhang, Ning & Zhuo, Zhenyu & Kang, Chongqing & Kirschen, Daniel, 2018. "Mixed-integer linear programming-based optimal configuration planning for energy hub: Starting from scratch," Applied Energy, Elsevier, vol. 210(C), pages 1141-1150.
    18. Barbón, A. & Bayón-Cueli, C. & Bayón, L. & Carreira-Fontao, V., 2022. "A methodology for an optimal design of ground-mounted photovoltaic power plants," Applied Energy, Elsevier, vol. 314(C).
    19. Zhang, Jinhua & Yan, Jie & Infield, David & Liu, Yongqian & Lien, Fue-sang, 2019. "Short-term forecasting and uncertainty analysis of wind turbine power based on long short-term memory network and Gaussian mixture model," Applied Energy, Elsevier, vol. 241(C), pages 229-244.
    20. Lu, Xinhui & Liu, Zhaoxi & Ma, Li & Wang, Lingfeng & Zhou, Kaile & Feng, Nanping, 2020. "A robust optimization approach for optimal load dispatch of community energy hub," Applied Energy, Elsevier, vol. 259(C).
    21. Tian, Hang & Zhao, Haoran & Liu, Chunyang & Chen, Jian & Wu, Qiuwei & Terzija, Vladimir, 2022. "A dual-driven linear modeling approach for multiple energy flow calculation in electricity–heat system," Applied Energy, Elsevier, vol. 314(C).
    22. Xu, Xiao & Hu, Weihao & Liu, Wen & Du, Yuefang & Huang, Rui & Huang, Qi & Chen, Zhe, 2021. "Look-ahead risk-constrained scheduling for an energy hub integrated with renewable energy," Applied Energy, Elsevier, vol. 297(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    2. Zhang, Xiaoshun & Guo, Zhengxun & Pan, Feng & Yang, Yuyao & Li, Chuansheng, 2023. "Dynamic carbon emission factor based interactive control of distribution network by a generalized regression neural network assisted optimization," Energy, Elsevier, vol. 283(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Qiu, Dawei & Xue, Juxing & Zhang, Tingqi & Wang, Jianhong & Sun, Mingyang, 2023. "Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading," Applied Energy, Elsevier, vol. 333(C).
    2. Aslani, Mehrdad & Mashayekhi, Mehdi & Hashemi-Dezaki, Hamed & Ketabi, Abbas, 2022. "Robust optimal operation of energy hub incorporating integrated thermal and electrical demand response programs under various electric vehicle charging modes," Applied Energy, Elsevier, vol. 321(C).
    3. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    4. Najafi, Arsalan & Pourakbari-Kasmaei, Mahdi & Jasinski, Michal & Lehtonen, Matti & Leonowicz, Zbigniew, 2021. "A hybrid decentralized stochastic-robust model for optimal coordination of electric vehicle aggregator and energy hub entities," Applied Energy, Elsevier, vol. 304(C).
    5. Xie, Shiwei & Zheng, Jieyun & Hu, Zhijian & Wang, Jueying & Chen, Yuwei, 2020. "Urban multi-energy network optimization: An enhanced model using a two-stage bound-tightening approach," Applied Energy, Elsevier, vol. 277(C).
    6. Saberi-Beglar, Kasra & Zare, Kazem & Seyedi, Heresh & Marzband, Mousa & Nojavan, Sayyad, 2023. "Risk-embedded scheduling of a CCHP integrated with electric vehicle parking lot in a residential energy hub considering flexible thermal and electrical loads," Applied Energy, Elsevier, vol. 329(C).
    7. Roth, Jonathan & Martin, Amory & Miller, Clayton & Jain, Rishee K., 2020. "SynCity: Using open data to create a synthetic city of hourly building energy estimates by integrating data-driven and physics-based methods," Applied Energy, Elsevier, vol. 280(C).
    8. Jonathan Roth & Jayashree Chadalawada & Rishee K. Jain & Clayton Miller, 2021. "Uncertainty Matters: Bayesian Probabilistic Forecasting for Residential Smart Meter Prediction, Segmentation, and Behavioral Measurement and Verification," Energies, MDPI, vol. 14(5), pages 1-22, March.
    9. Seyfi, Mohammad & Mehdinejad, Mehdi & Mohammadi-Ivatloo, Behnam & Shayanfar, Heidarali, 2022. "Deep learning-based scheduling of virtual energy hubs with plug-in hybrid compressed natural gas-electric vehicles," Applied Energy, Elsevier, vol. 321(C).
    10. Vahid-Ghavidel, Morteza & Shafie-khah, Miadreza & Javadi, Mohammad S. & Santos, Sérgio F. & Gough, Matthew & Quijano, Darwin A. & Catalao, Joao P.S., 2023. "Hybrid IGDT-stochastic self-scheduling of a distributed energy resources aggregator in a multi-energy system," Energy, Elsevier, vol. 265(C).
    11. Akbari, Ehsan & Mousavi Shabestari, Seyed Farzin & Pirouzi, Sasan & Jadidoleslam, Morteza, 2023. "Network flexibility regulation by renewable energy hubs using flexibility pricing-based energy management," Renewable Energy, Elsevier, vol. 206(C), pages 295-308.
    12. Christina Papadimitriou & Marialaura Di Somma & Chrysanthos Charalambous & Martina Caliano & Valeria Palladino & Andrés Felipe Cortés Borray & Amaia González-Garrido & Nerea Ruiz & Giorgio Graditi, 2023. "A Comprehensive Review of the Design and Operation Optimization of Energy Hubs and Their Interaction with the Markets and External Networks," Energies, MDPI, vol. 16(10), pages 1-46, May.
    13. Gan, Wei & Yan, Mingyu & Yao, Wei & Guo, Jianbo & Ai, Xiaomeng & Fang, Jiakun & Wen, Jinyu, 2021. "Decentralized computation method for robust operation of multi-area joint regional-district integrated energy systems with uncertain wind power," Applied Energy, Elsevier, vol. 298(C).
    14. Gržanić, M. & Capuder, T. & Zhang, N. & Huang, W., 2022. "Prosumers as active market participants: A systematic review of evolution of opportunities, models and challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 154(C).
    15. Gao, Fang & Xu, Zidong & Yin, Linfei, 2024. "Bayesian deep neural networks for spatio-temporal probabilistic optimal power flow with multi-source renewable energy," Applied Energy, Elsevier, vol. 353(PA).
    16. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    17. Mittelviefhaus, Moritz & Pareschi, Giacomo & Allan, James & Georges, Gil & Boulouchos, Konstantinos, 2021. "Optimal investment and scheduling of residential multi-energy systems including electric mobility: A cost-effective approach to climate change mitigation," Applied Energy, Elsevier, vol. 301(C).
    18. Cai, Qingsen & Luo, XingQi & Wang, Peng & Gao, Chunyang & Zhao, Peiyu, 2022. "Hybrid model-driven and data-driven control method based on machine learning algorithm in energy hub and application," Applied Energy, Elsevier, vol. 305(C).
    19. Yu, Binbin & Li, Jianjing & Liu, Che & Sun, Bo, 2022. "A novel short-term electrical load forecasting framework with intelligent feature engineering," Applied Energy, Elsevier, vol. 327(C).
    20. Najafi, Arsalan & Pourakbari-Kasmaei, Mahdi & Jasinski, Michal & Contreras, Javier & Lehtonen, Matti & Leonowicz, Zbigniew, 2022. "The role of EV based peer-to-peer transactive energy hubs in distribution network optimization," Applied Energy, Elsevier, vol. 319(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:339:y:2023:i:c:s0306261923002660. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.