IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v280y2023ics0360544223015761.html
   My bibliography  Save this article

Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning

Author

Listed:
  • Wang, Yijian
  • Cui, Yang
  • Li, Yang
  • Xu, Yang

Abstract

Achieving the economical and stable operation of Multi-microgrids (MMG) systems is vital. However, there are still some challenging problems to be solved. Firstly, from the perspective of stable operation, it is necessary to minimize the energy fluctuation of the main grid. Secondly, the characteristics of energy conversion equipment need to be considered. Finally, privacy protection while reducing the operating cost of an MMG system is crucial. To address these challenges, a Data-driven strategy for MMG systems with Shared Energy Storage (SES) is proposed. In this paper, the Mixed-Attention is applied to fit the conditions of the equipment, and Multi-Agent Soft Actor-Critic(MA-SAC) , Multi-Agent Win or Learn Fast Policy Hill-Climbing (MA-WoLF-PHC) are proposed to solve the partially observable dynamic stochastic game problem. By testing the operation data of the MMG system in Northwest China, following conclusions are drawn: the R-Square (R2) values of results reach 0.999, indicating the neural network effectively models the nonlinear conditions. The proposed MMG system framework can reduce energy fluctuations in the main grid by 1746.5 kW in 24 h and achieve a cost reduction of 16.21% in the test. Finally, the superiority of the proposed algorithms is verified through their fast convergence speed and excellent optimization performance.

Suggested Citation

  • Wang, Yijian & Cui, Yang & Li, Yang & Xu, Yang, 2023. "Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning," Energy, Elsevier, vol. 280(C).
    • Handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761
    DOI: 10.1016/j.energy.2023.128182
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223015761
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.128182?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Vitale, F. & Rispoli, N. & Sorrentino, M. & Rosen, M.A. & Pianese, C., 2021. "On the use of dynamic programming for optimal energy management of grid-connected reversible solid oxide cell-based renewable microgrids," Energy, Elsevier, vol. 225(C).
    2. Abunima, Hamza & Park, Woan-Ho & Glick, Mark B. & Kim, Yun-Su, 2022. "Two-Stage stochastic optimization for operating a Renewable-Based Microgrid," Applied Energy, Elsevier, vol. 325(C).
    3. Bischi, Aldo & Taccari, Leonardo & Martelli, Emanuele & Amaldi, Edoardo & Manzolini, Giampaolo & Silva, Paolo & Campanari, Stefano & Macchi, Ennio, 2014. "A detailed MILP optimization model for combined cooling, heat and power system operation planning," Energy, Elsevier, vol. 74(C), pages 12-26.
    4. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    5. Jiang, Jianhua & Zhou, Renjie & Xu, Hao & Wang, Hao & Wu, Ping & Wang, Zhuo & Li, Jian, 2022. "Optimal sizing, operation strategy and case study of a grid-connected solid oxide fuel cell microgrid," Applied Energy, Elsevier, vol. 307(C).
    6. Wang, Zhaoqi & Zhang, Lu & Tang, Wei & Chen, Ying & Shen, Chen, 2022. "Equilibrium allocation strategy of multiple ESSs considering the economics and restoration capability in DNs," Applied Energy, Elsevier, vol. 306(PA).
    7. Hu, Mian & Wang, Yan-Wu & Xiao, Jiang-Wen & Lin, Xiangning, 2019. "Multi-energy management with hierarchical distributed multi-scale strategy for pelagic islanded microgrid clusters," Energy, Elsevier, vol. 185(C), pages 910-921.
    8. Stoppato, Anna & Cavazzini, Giovanna & Ardizzon, Guido & Rossetti, Antonio, 2014. "A PSO (particle swarm optimization)-based model for the optimal management of a small PV(Photovoltaic)-pump hydro energy storage in a rural dry area," Energy, Elsevier, vol. 76(C), pages 168-174.
    9. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    10. dos Santos Neto, Pedro J. & Barros, Tárcio A.S. & Silveira, Joao P.C. & Ruppert Filho, Ernesto & Vasquez, Juan C. & Guerrero, Josep M., 2020. "Power management techniques for grid-connected DC microgrids: A comparative evaluation," Applied Energy, Elsevier, vol. 269(C).
    11. Nawaz, Arshad & Zhou, Min & Wu, Jing & Long, Chengnian, 2022. "A comprehensive review on energy management, demand response, and coordination schemes utilization in multi-microgrids network," Applied Energy, Elsevier, vol. 323(C).
    12. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    13. Zhao, Bo & Xue, Meidong & Zhang, Xuesong & Wang, Caisheng & Zhao, Junhui, 2015. "An MAS based energy management system for a stand-alone microgrid at high altitude," Applied Energy, Elsevier, vol. 143(C), pages 251-261.
    14. Nelson, James & Johnson, Nathan G. & Fahy, Kelsey & Hansen, Timothy A., 2020. "Statistical development of microgrid resilience during islanding operations," Applied Energy, Elsevier, vol. 279(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roldán-Blay, Carlos & Escrivá-Escrivá, Guillermo & Roldán-Porta, Carlos & Dasí-Crespo, Daniel, 2023. "Optimal sizing and design of renewable power plants in rural microgrids using multi-objective particle swarm optimization and branch and bound methods," Energy, Elsevier, vol. 284(C).
    2. Song, Xiaoling & Wu, Han & Zhang, Huqing & Guo, Jianxin & Zhang, Zhe & Peña-Mora, Feniosky, 2025. "Can retail electricity pricing promote microgrid operators to leverage shared energy storage services among internal aggregators?," Energy, Elsevier, vol. 314(C).
    3. Zhou, Wenbing & Chi, Yuanying & Tang, Songlin & Hu, Yu & Wang, Zhengzao & Zhang, Xufeng, 2024. "The evolution analysis of low-carbon power transition strategies and carbon emission decoupling based on carbon neutrality target," Energy, Elsevier, vol. 311(C).
    4. Peng Liu & Tieyan Zhang & Furui Tian & Yun Teng & Miaodong Yang, 2024. "Hybrid Decision Support Framework for Energy Scheduling Using Stochastic Optimization and Cooperative Game Theory," Energies, MDPI, vol. 17(24), pages 1-20, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wilson Pavon & Esteban Inga & Silvio Simani & Maddalena Nonato, 2021. "A Review on Optimal Control for the Smart Grid Electrical Substation Enhancing Transition Stability," Energies, MDPI, vol. 14(24), pages 1-15, December.
    2. Mahmud, Sadab & Ponkiya, Binaka & Katikaneni, Sravya & Pandey, Srijana & Mattimadugu, Kranthikiran & Yi, Zonggen & Walker, Victor & Wang, Congjian & Westover, Tyler & Javaid, Ahmad Y. & Heben, Michael, 2024. "Design and optimization of a modular hydrogen-based integrated energy system to maximize revenue via nuclear-renewable sources," Energy, Elsevier, vol. 313(C).
    3. He, Jiaming & Tan, Qinliang & Lv, Hanyu, 2025. "Data-driven climate resilience assessment for distributed energy systems using diffusion transformer and polynomial expansions," Applied Energy, Elsevier, vol. 380(C).
    4. Qi, Yunying & Xu, Xiao & Liu, Youbo & Pan, Li & Liu, Junyong & Hu, Weihao, 2024. "Intelligent energy management for an on-grid hydrogen refueling station based on dueling double deep Q network algorithm with NoisyNet," Renewable Energy, Elsevier, vol. 222(C).
    5. Yang, Ting & Xu, Zheming & Ji, Shijie & Liu, Guoliang & Li, Xinhong & Kong, Haibo, 2025. "Cooperative optimal dispatch of multi-microgrids for low carbon economy based on personalized federated reinforcement learning," Applied Energy, Elsevier, vol. 378(PA).
    6. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    7. Liu, Zhiyong & Zhang, Yuxian & Ning, Yi, 2025. "Emergency mobile energy storage optimal allocation in microgrid-integrated distribution networks considering economic and resilience benefits," Energy, Elsevier, vol. 322(C).
    8. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    9. Tan, Mao & Li, Zibin & Su, Yongxin & Ren, Yuling & Wang, Ling & Wang, Rui, 2024. "Dual time-scale robust optimization for energy management of distributed energy community considering source-load uncertainty," Renewable Energy, Elsevier, vol. 226(C).
    10. Yang, Jingxian & Liu, Junyong & Qiu, Gao & Liu, Jichun & Jawad, Shafqat & Zhang, Shuai, 2023. "A spatio-temporality-enabled parallel multi-agent-based real-time dynamic dispatch for hydro-PV-PHS integrated power system," Energy, Elsevier, vol. 278(PB).
    11. Duan, Jiandong & Tian, Qinxing & Liu, Fan & Xia, Yerui & Gao, Qi, 2024. "Optimal scheduling strategy with integrated demand response based on stepped incentive mechanism for integrated electricity-gas energy system," Energy, Elsevier, vol. 313(C).
    12. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    13. Fouad Boutros & Moustapha Doumiati & Jean-Christophe Olivier & Imad Mougharbel & Hadi Kanaan, 2024. "Optimal Placement of Multiple Sources in a Mesh-Type DC Microgrid Using Dijkstra’s Algorithm," Energies, MDPI, vol. 17(14), pages 1-18, July.
    14. Mousavi, Navid & Kothapalli, Ganesh & Habibi, Daryoush & Das, Choton K. & Baniasadi, Ali, 2020. "A novel photovoltaic-pumped hydro storage microgrid applicable to rural areas," Applied Energy, Elsevier, vol. 262(C).
    15. Guelpa, Elisa & Bischi, Aldo & Verda, Vittorio & Chertkov, Michael & Lund, Henrik, 2019. "Towards future infrastructures for sustainable multi-energy systems: A review," Energy, Elsevier, vol. 184(C), pages 2-21.
    16. Jimmy Gallegos & Paul Arévalo & Christian Montaleza & Francisco Jurado, 2024. "Sustainable Electrification—Advances and Challenges in Electrical-Distribution Networks: A Review," Sustainability, MDPI, vol. 16(2), pages 1-33, January.
    17. Rodriguez, Mauricio & Arcos-Aviles, Diego & Guinjoan, Francesc, 2024. "Simple fuzzy logic-based energy management for power exchange in isolated multi-microgrid systems: A case study in a remote community in the Amazon region of Ecuador," Applied Energy, Elsevier, vol. 357(C).
    18. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    19. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    20. Kang, Ligai & Yang, Junhong & An, Qingsong & Deng, Shuai & Zhao, Jun & Wang, Hui & Li, Zelin, 2017. "Effects of load following operational strategy on CCHP system with an auxiliary ground source heat pump considering carbon tax and electricity feed in tariff," Applied Energy, Elsevier, vol. 194(C), pages 454-466.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.