IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v280y2023ics0360544223015761.html
   My bibliography  Save this article

Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning

Author

Listed:
  • Wang, Yijian
  • Cui, Yang
  • Li, Yang
  • Xu, Yang

Abstract

Achieving the economical and stable operation of Multi-microgrids (MMG) systems is vital. However, there are still some challenging problems to be solved. Firstly, from the perspective of stable operation, it is necessary to minimize the energy fluctuation of the main grid. Secondly, the characteristics of energy conversion equipment need to be considered. Finally, privacy protection while reducing the operating cost of an MMG system is crucial. To address these challenges, a Data-driven strategy for MMG systems with Shared Energy Storage (SES) is proposed. In this paper, the Mixed-Attention is applied to fit the conditions of the equipment, and Multi-Agent Soft Actor-Critic(MA-SAC) , Multi-Agent Win or Learn Fast Policy Hill-Climbing (MA-WoLF-PHC) are proposed to solve the partially observable dynamic stochastic game problem. By testing the operation data of the MMG system in Northwest China, following conclusions are drawn: the R-Square (R2) values of results reach 0.999, indicating the neural network effectively models the nonlinear conditions. The proposed MMG system framework can reduce energy fluctuations in the main grid by 1746.5 kW in 24 h and achieve a cost reduction of 16.21% in the test. Finally, the superiority of the proposed algorithms is verified through their fast convergence speed and excellent optimization performance.

Suggested Citation

  • Wang, Yijian & Cui, Yang & Li, Yang & Xu, Yang, 2023. "Collaborative optimization of multi-microgrids system with shared energy storage based on multi-agent stochastic game and reinforcement learning," Energy, Elsevier, vol. 280(C).
    • Handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761
    DOI: 10.1016/j.energy.2023.128182
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223015761
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.128182?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Nawaz, Arshad & Zhou, Min & Wu, Jing & Long, Chengnian, 2022. "A comprehensive review on energy management, demand response, and coordination schemes utilization in multi-microgrids network," Applied Energy, Elsevier, vol. 323(C).
    2. Jiang, Jianhua & Zhou, Renjie & Xu, Hao & Wang, Hao & Wu, Ping & Wang, Zhuo & Li, Jian, 2022. "Optimal sizing, operation strategy and case study of a grid-connected solid oxide fuel cell microgrid," Applied Energy, Elsevier, vol. 307(C).
    3. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    4. Wang, Zhaoqi & Zhang, Lu & Tang, Wei & Chen, Ying & Shen, Chen, 2022. "Equilibrium allocation strategy of multiple ESSs considering the economics and restoration capability in DNs," Applied Energy, Elsevier, vol. 306(PA).
    5. Hu, Mian & Wang, Yan-Wu & Xiao, Jiang-Wen & Lin, Xiangning, 2019. "Multi-energy management with hierarchical distributed multi-scale strategy for pelagic islanded microgrid clusters," Energy, Elsevier, vol. 185(C), pages 910-921.
    6. dos Santos Neto, Pedro J. & Barros, Tárcio A.S. & Silveira, Joao P.C. & Ruppert Filho, Ernesto & Vasquez, Juan C. & Guerrero, Josep M., 2020. "Power management techniques for grid-connected DC microgrids: A comparative evaluation," Applied Energy, Elsevier, vol. 269(C).
    7. Vitale, F. & Rispoli, N. & Sorrentino, M. & Rosen, M.A. & Pianese, C., 2021. "On the use of dynamic programming for optimal energy management of grid-connected reversible solid oxide cell-based renewable microgrids," Energy, Elsevier, vol. 225(C).
    8. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    9. Stoppato, Anna & Cavazzini, Giovanna & Ardizzon, Guido & Rossetti, Antonio, 2014. "A PSO (particle swarm optimization)-based model for the optimal management of a small PV(Photovoltaic)-pump hydro energy storage in a rural dry area," Energy, Elsevier, vol. 76(C), pages 168-174.
    10. Zhao, Bo & Xue, Meidong & Zhang, Xuesong & Wang, Caisheng & Zhao, Junhui, 2015. "An MAS based energy management system for a stand-alone microgrid at high altitude," Applied Energy, Elsevier, vol. 143(C), pages 251-261.
    11. Abunima, Hamza & Park, Woan-Ho & Glick, Mark B. & Kim, Yun-Su, 2022. "Two-Stage stochastic optimization for operating a Renewable-Based Microgrid," Applied Energy, Elsevier, vol. 325(C).
    12. Nelson, James & Johnson, Nathan G. & Fahy, Kelsey & Hansen, Timothy A., 2020. "Statistical development of microgrid resilience during islanding operations," Applied Energy, Elsevier, vol. 279(C).
    13. Bischi, Aldo & Taccari, Leonardo & Martelli, Emanuele & Amaldi, Edoardo & Manzolini, Giampaolo & Silva, Paolo & Campanari, Stefano & Macchi, Ennio, 2014. "A detailed MILP optimization model for combined cooling, heat and power system operation planning," Energy, Elsevier, vol. 74(C), pages 12-26.
    14. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roldán-Blay, Carlos & Escrivá-Escrivá, Guillermo & Roldán-Porta, Carlos & Dasí-Crespo, Daniel, 2023. "Optimal sizing and design of renewable power plants in rural microgrids using multi-objective particle swarm optimization and branch and bound methods," Energy, Elsevier, vol. 284(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wilson Pavon & Esteban Inga & Silvio Simani & Maddalena Nonato, 2021. "A Review on Optimal Control for the Smart Grid Electrical Substation Enhancing Transition Stability," Energies, MDPI, vol. 14(24), pages 1-15, December.
    2. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Tan, Mao & Li, Zibin & Su, Yongxin & Ren, Yuling & Wang, Ling & Wang, Rui, 2024. "Dual time-scale robust optimization for energy management of distributed energy community considering source-load uncertainty," Renewable Energy, Elsevier, vol. 226(C).
    5. Yang, Jingxian & Liu, Junyong & Qiu, Gao & Liu, Jichun & Jawad, Shafqat & Zhang, Shuai, 2023. "A spatio-temporality-enabled parallel multi-agent-based real-time dynamic dispatch for hydro-PV-PHS integrated power system," Energy, Elsevier, vol. 278(PB).
    6. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    7. Fouad Boutros & Moustapha Doumiati & Jean-Christophe Olivier & Imad Mougharbel & Hadi Kanaan, 2024. "Optimal Placement of Multiple Sources in a Mesh-Type DC Microgrid Using Dijkstra’s Algorithm," Energies, MDPI, vol. 17(14), pages 1-18, July.
    8. Mousavi, Navid & Kothapalli, Ganesh & Habibi, Daryoush & Das, Choton K. & Baniasadi, Ali, 2020. "A novel photovoltaic-pumped hydro storage microgrid applicable to rural areas," Applied Energy, Elsevier, vol. 262(C).
    9. Guelpa, Elisa & Bischi, Aldo & Verda, Vittorio & Chertkov, Michael & Lund, Henrik, 2019. "Towards future infrastructures for sustainable multi-energy systems: A review," Energy, Elsevier, vol. 184(C), pages 2-21.
    10. Jimmy Gallegos & Paul Arévalo & Christian Montaleza & Francisco Jurado, 2024. "Sustainable Electrification—Advances and Challenges in Electrical-Distribution Networks: A Review," Sustainability, MDPI, vol. 16(2), pages 1-33, January.
    11. Rodriguez, Mauricio & Arcos-Aviles, Diego & Guinjoan, Francesc, 2024. "Simple fuzzy logic-based energy management for power exchange in isolated multi-microgrid systems: A case study in a remote community in the Amazon region of Ecuador," Applied Energy, Elsevier, vol. 357(C).
    12. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    13. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    14. Kang, Ligai & Yang, Junhong & An, Qingsong & Deng, Shuai & Zhao, Jun & Wang, Hui & Li, Zelin, 2017. "Effects of load following operational strategy on CCHP system with an auxiliary ground source heat pump considering carbon tax and electricity feed in tariff," Applied Energy, Elsevier, vol. 194(C), pages 454-466.
    15. Chen, Lei & Jiang, Yuqi & Zheng, Shencong & Deng, Xinyi & Chen, Hongkun & Islam, Md. Rabiul, 2023. "A two-layer optimal configuration approach of energy storage systems for resilience enhancement of active distribution networks," Applied Energy, Elsevier, vol. 350(C).
    16. Tan, Bifei & Lin, Zhenjia & Zheng, Xiaodong & Xiao, Fu & Wu, Qiuwei & Yan, Jinyue, 2023. "Distributionally robust energy management for multi-microgrids with grid-interactive EVs considering the multi-period coupling effect of user behaviors," Applied Energy, Elsevier, vol. 350(C).
    17. Xiaofeng Liu & Shijun Wang & Jiawen Sun, 2018. "Energy Management for Community Energy Network with CHP Based on Cooperative Game," Energies, MDPI, vol. 11(5), pages 1-18, April.
    18. Cosgrove, Paul & Roulstone, Tony & Zachary, Stan, 2023. "Intermittency and periodicity in net-zero renewable energy systems with storage," Renewable Energy, Elsevier, vol. 212(C), pages 299-307.
    19. Shen, Feifei & Zhao, Liang & Du, Wenli & Zhong, Weimin & Qian, Feng, 2020. "Large-scale industrial energy systems optimization under uncertainty: A data-driven robust optimization approach," Applied Energy, Elsevier, vol. 259(C).
    20. Zhang, Bin & Hu, Weihao & Ghias, Amer M.Y.M. & Xu, Xiao & Chen, Zhe, 2022. "Multi-agent deep reinforcement learning-based coordination control for grid-aware multi-buildings," Applied Energy, Elsevier, vol. 328(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:280:y:2023:i:c:s0360544223015761. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.