IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v280y2023ics0360544223015761.html

Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning

Author

Listed:
  • Wang, Yijian
  • Cui, Yang
  • Li, Yang
  • Xu, Yang

Abstract

Achieving the economical and stable operation of Multi-microgrids (MMG) systems is vital. However, there are still some challenging problems to be solved. Firstly, from the perspective of stable operation, it is necessary to minimize the energy fluctuation of the main grid. Secondly, the characteristics of energy conversion equipment need to be considered. Finally, privacy protection while reducing the operating cost of an MMG system is crucial. To address these challenges, a Data-driven strategy for MMG systems with Shared Energy Storage (SES) is proposed. In this paper, the Mixed-Attention is applied to fit the conditions of the equipment, and Multi-Agent Soft Actor-Critic(MA-SAC) , Multi-Agent Win or Learn Fast Policy Hill-Climbing (MA-WoLF-PHC) are proposed to solve the partially observable dynamic stochastic game problem. By testing the operation data of the MMG system in Northwest China, following conclusions are drawn: the R-Square (R2) values of results reach 0.999, indicating the neural network effectively models the nonlinear conditions. The proposed MMG system framework can reduce energy fluctuations in the main grid by 1746.5 kW in 24 h and achieve a cost reduction of 16.21% in the test. Finally, the superiority of the proposed algorithms is verified through their fast convergence speed and excellent optimization performance.

Suggested Citation

  • Wang, Yijian & Cui, Yang & Li, Yang & Xu, Yang, 2023. "Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning," Energy, Elsevier, vol. 280(C).
    • Handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761
    DOI: 10.1016/j.energy.2023.128182
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223015761
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.128182?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Nawaz, Arshad & Zhou, Min & Wu, Jing & Long, Chengnian, 2022. "A comprehensive review on energy management, demand response, and coordination schemes utilization in multi-microgrids network," Applied Energy, Elsevier, vol. 323(C).
    2. Jiang, Jianhua & Zhou, Renjie & Xu, Hao & Wang, Hao & Wu, Ping & Wang, Zhuo & Li, Jian, 2022. "Optimal sizing, operation strategy and case study of a grid-connected solid oxide fuel cell microgrid," Applied Energy, Elsevier, vol. 307(C).
    3. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    4. Wang, Zhaoqi & Zhang, Lu & Tang, Wei & Chen, Ying & Shen, Chen, 2022. "Equilibrium allocation strategy of multiple ESSs considering the economics and restoration capability in DNs," Applied Energy, Elsevier, vol. 306(PA).
    5. Hu, Mian & Wang, Yan-Wu & Xiao, Jiang-Wen & Lin, Xiangning, 2019. "Multi-energy management with hierarchical distributed multi-scale strategy for pelagic islanded microgrid clusters," Energy, Elsevier, vol. 185(C), pages 910-921.
    6. dos Santos Neto, Pedro J. & Barros, Tárcio A.S. & Silveira, Joao P.C. & Ruppert Filho, Ernesto & Vasquez, Juan C. & Guerrero, Josep M., 2020. "Power management techniques for grid-connected DC microgrids: A comparative evaluation," Applied Energy, Elsevier, vol. 269(C).
    7. Vitale, F. & Rispoli, N. & Sorrentino, M. & Rosen, M.A. & Pianese, C., 2021. "On the use of dynamic programming for optimal energy management of grid-connected reversible solid oxide cell-based renewable microgrids," Energy, Elsevier, vol. 225(C).
    8. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    9. Stoppato, Anna & Cavazzini, Giovanna & Ardizzon, Guido & Rossetti, Antonio, 2014. "A PSO (particle swarm optimization)-based model for the optimal management of a small PV(Photovoltaic)-pump hydro energy storage in a rural dry area," Energy, Elsevier, vol. 76(C), pages 168-174.
    10. Zhao, Bo & Xue, Meidong & Zhang, Xuesong & Wang, Caisheng & Zhao, Junhui, 2015. "An MAS based energy management system for a stand-alone microgrid at high altitude," Applied Energy, Elsevier, vol. 143(C), pages 251-261.
    11. Abunima, Hamza & Park, Woan-Ho & Glick, Mark B. & Kim, Yun-Su, 2022. "Two-Stage stochastic optimization for operating a Renewable-Based Microgrid," Applied Energy, Elsevier, vol. 325(C).
    12. Nelson, James & Johnson, Nathan G. & Fahy, Kelsey & Hansen, Timothy A., 2020. "Statistical development of microgrid resilience during islanding operations," Applied Energy, Elsevier, vol. 279(C).
    13. Bischi, Aldo & Taccari, Leonardo & Martelli, Emanuele & Amaldi, Edoardo & Manzolini, Giampaolo & Silva, Paolo & Campanari, Stefano & Macchi, Ennio, 2014. "A detailed MILP optimization model for combined cooling, heat and power system operation planning," Energy, Elsevier, vol. 74(C), pages 12-26.
    14. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roldán-Blay, Carlos & Escrivá-Escrivá, Guillermo & Roldán-Porta, Carlos & Dasí-Crespo, Daniel, 2023. "Optimal sizing and design of renewable power plants in rural microgrids using multi-objective particle swarm optimization and branch and bound methods," Energy, Elsevier, vol. 284(C).
    2. Liu, Jicheng & Song, Yanan & Xue, Xiaojun & Duan, Bingfan & Hadi, Dhaif Allah, 2025. "Value evaluation model study on shared energy storage adapted to the needs of new power system," Energy, Elsevier, vol. 330(C).
    3. Song, Xiaoling & Wu, Han & Zhang, Huqing & Guo, Jianxin & Zhang, Zhe & Peña-Mora, Feniosky, 2025. "Can retail electricity pricing promote microgrid operators to leverage shared energy storage services among internal aggregators?," Energy, Elsevier, vol. 314(C).
    4. Zhou, Wenbing & Chi, Yuanying & Tang, Songlin & Hu, Yu & Wang, Zhengzao & Zhang, Xufeng, 2024. "The evolution analysis of low-carbon power transition strategies and carbon emission decoupling based on carbon neutrality target," Energy, Elsevier, vol. 311(C).
    5. Peng Liu & Tieyan Zhang & Furui Tian & Yun Teng & Miaodong Yang, 2024. "Hybrid Decision Support Framework for Energy Scheduling Using Stochastic Optimization and Cooperative Game Theory," Energies, MDPI, vol. 17(24), pages 1-20, December.
    6. Mahmud, Sakib & Sayed, Aya Nabil & Himeur, Yassine & Nhlabatsi, Armstrong & Bensaali, Faycal, 2026. "A comprehensive review of deep reinforcement learning applications from centralized power generation to modern energy internet frameworks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PE).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wilson Pavon & Esteban Inga & Silvio Simani & Maddalena Nonato, 2021. "A Review on Optimal Control for the Smart Grid Electrical Substation Enhancing Transition Stability," Energies, MDPI, vol. 14(24), pages 1-15, December.
    2. Mahmud, Sadab & Ponkiya, Binaka & Katikaneni, Sravya & Pandey, Srijana & Mattimadugu, Kranthikiran & Yi, Zonggen & Walker, Victor & Wang, Congjian & Westover, Tyler & Javaid, Ahmad Y. & Heben, Michael, 2024. "Design and optimization of a modular hydrogen-based integrated energy system to maximize revenue via nuclear-renewable sources," Energy, Elsevier, vol. 313(C).
    3. He, Jiaming & Tan, Qinliang & Lv, Hanyu, 2025. "Data-driven climate resilience assessment for distributed energy systems using diffusion transformer and polynomial expansions," Applied Energy, Elsevier, vol. 380(C).
    4. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    5. Tan, Mao & Li, Zibin & Su, Yongxin & Ren, Yuling & Wang, Ling & Wang, Rui, 2024. "Dual time-scale robust optimization for energy management of distributed energy community considering source-load uncertainty," Renewable Energy, Elsevier, vol. 226(C).
    6. Qi, Yunying & Xu, Xiao & Liu, Youbo & Pan, Li & Liu, Junyong & Hu, Weihao, 2024. "Intelligent energy management for an on-grid hydrogen refueling station based on dueling double deep Q network algorithm with NoisyNet," Renewable Energy, Elsevier, vol. 222(C).
    7. Yang, Jingxian & Liu, Junyong & Qiu, Gao & Liu, Jichun & Jawad, Shafqat & Zhang, Shuai, 2023. "A spatio-temporality-enabled parallel multi-agent-based real-time dynamic dispatch for hydro-PV-PHS integrated power system," Energy, Elsevier, vol. 278(PB).
    8. Yang, Ting & Xu, Zheming & Ji, Shijie & Liu, Guoliang & Li, Xinhong & Kong, Haibo, 2025. "Cooperative optimal dispatch of multi-microgrids for low carbon economy based on personalized federated reinforcement learning," Applied Energy, Elsevier, vol. 378(PA).
    9. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    10. Liu, Zhiyong & Zhang, Yuxian & Ning, Yi, 2025. "Emergency mobile energy storage optimal allocation in microgrid-integrated distribution networks considering economic and resilience benefits," Energy, Elsevier, vol. 322(C).
    11. Li, Feng & Hou, Hui & Ni, Tian & Wang, Pingfan & Wang, Yingxiang & Li, Zhengmao & Pouresmaeil, Edris, 2025. "A novel deep reinforcement learning framework for optimal scheduling of low-carbon integrated energy systems considering battery degradation," Energy, Elsevier, vol. 341(C).
    12. Duan, Jiandong & Tian, Qinxing & Liu, Fan & Xia, Yerui & Gao, Qi, 2024. "Optimal scheduling strategy with integrated demand response based on stepped incentive mechanism for integrated electricity-gas energy system," Energy, Elsevier, vol. 313(C).
    13. Armghan, Hammad & Xu, Yinliang & Bai, Xiang & Ali, Naghmash & Chang, Xinyue & Xue, Yixun, 2024. "A tri-level control framework for carbon-aware multi-energy microgrid cluster considering shared hydrogen energy storage," Applied Energy, Elsevier, vol. 373(C).
    14. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    15. Zhang, Shuo & Wei, Ming & Li, Yingzi & Chen, Yuanli, 2025. "A Stackelberg-game based bi-level scheduling model of data center combined with shared energy storage considering price linkage and demand response," Energy, Elsevier, vol. 336(C).
    16. Fouad Boutros & Moustapha Doumiati & Jean-Christophe Olivier & Imad Mougharbel & Hadi Kanaan, 2024. "Optimal Placement of Multiple Sources in a Mesh-Type DC Microgrid Using Dijkstra’s Algorithm," Energies, MDPI, vol. 17(14), pages 1-18, July.
    17. Mousavi, Navid & Kothapalli, Ganesh & Habibi, Daryoush & Das, Choton K. & Baniasadi, Ali, 2020. "A novel photovoltaic-pumped hydro storage microgrid applicable to rural areas," Applied Energy, Elsevier, vol. 262(C).
    18. Hadi, Mojtaba & Elbouchikhi, Elhoussin & Zhou, Zhibin & Saim, Abdelhakim, 2026. "Optimal energy management in multi energy microgrid with combined heat and power system and demand side response integration," Renewable Energy, Elsevier, vol. 256(PE).
    19. Guelpa, Elisa & Bischi, Aldo & Verda, Vittorio & Chertkov, Michael & Lund, Henrik, 2019. "Towards future infrastructures for sustainable multi-energy systems: A review," Energy, Elsevier, vol. 184(C), pages 2-21.
    20. Jimmy Gallegos & Paul Arévalo & Christian Montaleza & Francisco Jurado, 2024. "Sustainable Electrification—Advances and Challenges in Electrical-Distribution Networks: A Review," Sustainability, MDPI, vol. 16(2), pages 1-33, January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.