IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v16y2023i7p3248-d1116390.html

Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm

Author

Listed:
  • Jiankai Gao

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

  • Yang Li

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

  • Bin Wang

    (State Grid Jining Power Supply Company, Jining 272000, China)

  • Haibo Wu

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

Abstract

The implementation of a multi-microgrid (MMG) system with multiple renewable energy sources enables the facilitation of electricity trading. To tackle the energy management problem of an MMG system, which consists of multiple renewable energy microgrids belonging to different operating entities, this paper proposes an MMG collaborative optimization scheduling model based on a multi-agent centralized training distributed execution framework. To enhance the generalization ability of dealing with various uncertainties, we also propose an improved multi-agent soft actor-critic (MASAC) algorithm, which facilitates energy transactions between multi-agents in MMG, and employs automated machine learning (AutoML) to optimize the MASAC hyperparameters to further improve the generalization of deep reinforcement learning (DRL). The test results demonstrate that the proposed method successfully achieves power complementarity between different entities and reduces the MMG system’s operating cost. Additionally, the proposal significantly outperforms other state-of-the-art reinforcement learning algorithms with better economy and higher calculation efficiency.

Suggested Citation

  • Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    • Handle: RePEc:gam:jeners:v:16:y:2023:i:7:p:3248-:d:1116390
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/16/7/3248/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/16/7/3248/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    2. Xie, Peilin & Tan, Sen & Bazmohammadi, Najmeh & Guerrero, Josep. M. & Vasquez, Juan. C. & Alcala, Jose Matas & Carreño, Jorge El Mariachet, 2022. "A distributed real-time power management scheme for shipboard zonal multi-microgrid system," Applied Energy, Elsevier, vol. 317(C).
    3. Wang, Yong & Wu, Yuankai & Tang, Yingjuan & Li, Qin & He, Hongwen, 2023. "Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 332(C).
    4. Hakimi, Seyed Mehdi & Hasankhani, Arezoo & Shafie-khah, Miadreza & Catalão, João P.S., 2021. "Stochastic planning of a multi-microgrid considering integration of renewable energy resources and real-time electricity market," Applied Energy, Elsevier, vol. 298(C).
    5. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    6. Parlikar, Anupam & Schott, Maximilian & Godse, Ketaki & Kucevic, Daniel & Jossen, Andreas & Hesse, Holger, 2023. "High-power electric vehicle charging: Low-carbon grid integration pathways with stationary lithium-ion battery systems and renewable generation," Applied Energy, Elsevier, vol. 333(C).
    7. Chen, Weidong & Wang, Junnan & Yu, Guanyi & Chen, Jiajia & Hu, Yumeng, 2022. "Research on day-ahead transactions between multi-microgrid based on cooperative game model," Applied Energy, Elsevier, vol. 316(C).
    8. Li, Yang & Bu, Fanjin & Li, Yuanzheng & Long, Chao, 2023. "Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 333(C).
    9. Park, Keonwoo & Moon, Ilkyeong, 2022. "Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid," Applied Energy, Elsevier, vol. 328(C).
    10. Soleimanzade, Mohammad Amin & Kumar, Amit & Sadrzadeh, Mohtada, 2022. "Novel data-driven energy management of a hybrid photovoltaic-reverse osmosis desalination system using deep reinforcement learning," Applied Energy, Elsevier, vol. 317(C).
    11. Li, Yang & Feng, Bo & Wang, Bin & Sun, Shuchao, 2022. "Joint planning of distributed generations and energy storage in active distribution networks: A Bi-Level programming approach," Energy, Elsevier, vol. 245(C).
    12. Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    13. Nawaz, Arshad & Wu, Jing & Ye, Jun & Dong, Yidi & Long, Chengnian, 2023. "Distributed MPC-based energy scheduling for islanded multi-microgrid considering battery degradation and cyclic life deterioration," Applied Energy, Elsevier, vol. 329(C).
    14. Li, Yang & Wang, Ruinong & Li, Yuanzheng & Zhang, Meng & Long, Chao, 2023. "Wind power forecasting considering data privacy protection: A federated deep reinforcement learning approach," Applied Energy, Elsevier, vol. 329(C).
    15. Li, Yang & Wang, Bin & Yang, Zhen & Li, Jiazheng & Chen, Chen, 2022. "Hierarchical stochastic scheduling of multi-community integrated energy systems in uncertain environments via Stackelberg game," Applied Energy, Elsevier, vol. 308(C).
    16. Zhang, Xizheng & Wang, Zeyu & Lu, Zhangyu, 2022. "Multi-objective load dispatch for microgrid with electric vehicles using modified gravitational search and particle swarm optimization algorithm," Applied Energy, Elsevier, vol. 306(PA).
    17. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    18. Kim, H.J. & Kim, M.K., 2023. "A novel deep learning-based forecasting model optimized by heuristic algorithm for energy management of microgrid," Applied Energy, Elsevier, vol. 332(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mahmud, Sakib & Sayed, Aya Nabil & Himeur, Yassine & Nhlabatsi, Armstrong & Bensaali, Faycal, 2026. "A comprehensive review of deep reinforcement learning applications from centralized power generation to modern energy internet frameworks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PE).
    2. Li, Xinyan & Wu, Nan, 2024. "A two-stage distributed robust optimal control strategy for energy collaboration in multi-regional integrated energy systems based on cooperative game," Energy, Elsevier, vol. 305(C).
    3. Li, Yang & Zhao, Bingsong & Li, Yuanzheng & Long, Chao & Li, Sen & Dong, Zhaoyang & Shahidehpour, Mohammad, 2025. "Safe-AutoSAC: AutoML-enhanced safe deep reinforcement learning for integrated energy system scheduling with multi-channel informer forecasting and electric vehicle demand response," Applied Energy, Elsevier, vol. 399(C).
    4. Li, Yang & Wang, Ruinong & Li, Yuanzheng & Zhang, Meng & Long, Chao, 2023. "Wind power forecasting considering data privacy protection: A federated deep reinforcement learning approach," Applied Energy, Elsevier, vol. 329(C).
    5. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    6. Li, Yang & Bu, Fanjin & Li, Yuanzheng & Long, Chao, 2023. "Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 333(C).
    7. Fan, Wei & Fan, Ying & Yao, Xing & Yi, Bowen & Jiang, Dalin & Wu, Lin, 2024. "Distributed transaction optimization model of multi-integrated energy systems based on nash negotiation," Renewable Energy, Elsevier, vol. 225(C).
    8. Han, Fengwu & Zeng, Jianfeng & Lin, Junjie & Zhao, Yunlong & Gao, Chong, 2023. "A stochastic hierarchical optimization and revenue allocation approach for multi-regional integrated energy systems based on cooperative games," Applied Energy, Elsevier, vol. 350(C).
    9. Liao, Chenxin & Miyata, Shohei & Qu, Ming & Akashi, Yasunori, 2025. "Year-round operational optimization of HVAC systems using hierarchical deep reinforcement learning for enhancing indoor air quality and reducing energy consumption," Applied Energy, Elsevier, vol. 390(C).
    10. Jani, Ali & Jadid, Shahram, 2023. "Two-stage energy scheduling framework for multi-microgrid system in market environment," Applied Energy, Elsevier, vol. 336(C).
    11. Jing, Wang & Aksoy, Muammer & Rahman, Md Arafatur & Alenezi, Ali H. & Deriche, Mohamed & Tao, Hai, 2026. "Multi-objective planning and optimal configuration of wind, solar, and energy storage in interconnected microgrid clusters using Vine Copula scenario generation and antlion optimization," Renewable Energy, Elsevier, vol. 256(PE).
    12. Wang, Chunling & Liu, Chunming & Chen, Jian & Zhang, Gaoyuan, 2024. "Cooperative planning of renewable energy generation and multi-timescale flexible resources in active distribution networks," Applied Energy, Elsevier, vol. 356(C).
    13. Tong, He & Chu, Liang & Wang, Zixu & Zhao, Di, 2025. "Adaptive Pulse-and-Glide for synergistic optimization of driving behavior and energy management in hybrid powertrain," Energy, Elsevier, vol. 330(C).
    14. Lin, Zheng & Lou, Suhua & Liu, Chun, 2026. "Energy storage capacity planning for cross-regional resource complementarity: a cooperative game framework," Applied Energy, Elsevier, vol. 406(C).
    15. Gao, Yuan & Hu, Zehuan & Matsunami, Yuki & Qu, Ming & Chen, Wei-An & Liu, Mingzhe, 2026. "Optimizing renewable energy systems with hybrid action space reinforcement learning: A case study on achieving net zero energy in Japan," Renewable Energy, Elsevier, vol. 256(PG).
    16. Xu, Xun & Shao, Zhenguo & Chen, Feixiong & Cheng, Guoyang, 2024. "Multi-game optimization operation strategy for integrated energy system considering spatiotemporal correlation of renewable energy," Energy, Elsevier, vol. 303(C).
    17. Wei Wei & Li Ye & Yi Fang & Yingchun Wang & Xi Chen & Zhenhua Li, 2023. "Optimal Allocation of Energy Storage Capacity in Microgrids Considering the Uncertainty of Renewable Energy Generation," Sustainability, MDPI, vol. 15(12), pages 1-17, June.
    18. Xu, Xun & Shao, Zhenguo & Chen, Feixiong & Cheng, Guoyang, 2025. "Stochastic robust optimization scheduling for integrated energy system cluster based on data-driven method," Applied Energy, Elsevier, vol. 400(C).
    19. Shu, Lei & Mo, Yunjeong & Zhao, Dong, 2024. "Energy retrofits for smart and connected communities: Scopes and technologies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 199(C).
    20. Zhang, Chao & Yin, Wanjun & Wen, Tao, 2024. "An advanced multi-objective collaborative scheduling strategy for large scale EV charging and discharging connected to the predictable wind power grid," Energy, Elsevier, vol. 287(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:16:y:2023:i:7:p:3248-:d:1116390. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager The email address of this maintainer does not seem to be valid anymore. Please ask MDPI Indexing Manager to update the entry or send us the correct address (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.