IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v349y2023ics0306261923010127.html

Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay

Author

Listed:
  • Guo, Guodong
  • Zhang, Mengfan
  • Gong, Yanfeng
  • Xu, Qianwen

Abstract

The increasing penetration of distributed renewable energy resources brings a great challenge for real-time voltage security of distribution grids. The paper proposes a safe multi-agent deep reinforcement learning (MADRL) algorithm for real-time control of inverter-based Volt-Var control (VVC) in distribution grids considering communication delay to minimize the network power loss, while maintaining the nodal voltages in a safe range. The multi-agent VVC is modeled as a constrained Markov game, which is solved by the MADRL algorithm. In the training stage, the safety projection is added to the combined policy to analytically solve an action correction formulation to promote more efficient and safe exploration. In the real-time decision-making stage, a state synchronization block is designed to impute the data under the latest timestamp as the input of the agents deployed in a distributed manner, to avoid instability caused by communication delay. The simulation results show that the proposed algorithm performs well in safe exploration, and also achieves better performance under communication delay.

Suggested Citation

  • Guo, Guodong & Zhang, Mengfan & Gong, Yanfeng & Xu, Qianwen, 2023. "Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay," Applied Energy, Elsevier, vol. 349(C).
    • Handle: RePEc:eee:appene:v:349:y:2023:i:c:s0306261923010127
    DOI: 10.1016/j.apenergy.2023.121648
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923010127
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.121648?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Aasim, & Singh, S.N. & Mohapatra, Abheejeet, 2019. "Repeated wavelet transform based ARIMA model for very short-term wind speed forecasting," Renewable Energy, Elsevier, vol. 136(C), pages 758-768.
    2. Lee, Xian Yeow & Sarkar, Soumik & Wang, Yubo, 2022. "A graph policy network approach for Volt-Var Control in power distribution systems," Applied Energy, Elsevier, vol. 323(C).
    3. Gao, Yuanqi & Yu, Nanpeng, 2022. "Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks," Applied Energy, Elsevier, vol. 313(C).
    4. Cao, Di & Zhao, Junbo & Hu, Weihao & Ding, Fei & Yu, Nanpeng & Huang, Qi & Chen, Zhe, 2022. "Model-free voltage control of active distribution system with PVs using surrogate model-based deep reinforcement learning," Applied Energy, Elsevier, vol. 306(PA).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yu, Peipei & Zhang, Hongcai & Song, Yonghua & Wang, Zhenyi & Dong, Huiyu & Ji, Liang, 2025. "Safe reinforcement learning for power system control: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 223(C).
    2. Dae-Sung Lee & Sung-Yong Son, 2024. "Weighted Average Ensemble-Based PV Forecasting in a Limited Environment with Missing Data of PV Power," Sustainability, MDPI, vol. 16(10), pages 1-17, May.
    3. Ahmadi, Mehrnaz & Aly, Hamed & Gu, Jason, 2026. "A comprehensive review of AI-driven approaches for smart grid stability and reliability," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PD).
    4. Xue, Lin & Zhang, Yao & Wang, Jianxue & Li, Haotian & Li, Fangshi, 2024. "Privacy-preserving multi-level co-regulation of VPPs via hierarchical safe deep reinforcement learning," Applied Energy, Elsevier, vol. 371(C).
    5. Ye, Tong & Huang, Yuping & Yang, Weijia & Cai, Guotian & Yang, Yuyao & Pan, Feng, 2025. "Safe multi-agent deep reinforcement learning for decentralized low-carbon operation in active distribution networks and multi-microgrids," Applied Energy, Elsevier, vol. 387(C).
    6. Muhammad Ikram & Daryoush Habibi & Asma Aziz, 2025. "Networked Multi-Agent Deep Reinforcement Learning Framework for the Provision of Ancillary Services in Hybrid Power Plants," Energies, MDPI, vol. 18(10), pages 1-34, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    2. Kabir, Farzana & Yu, Nanpeng & Gao, Yuanqi & Wang, Wenyu, 2023. "Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems," Applied Energy, Elsevier, vol. 335(C).
    3. Zhang, Ziqi & Li, Peng & Ji, Haoran & Zhao, Jinli & Xi, Wei & Wu, Jianzhong & Wang, Chengshan, 2024. "Combined central-local voltage control of inverter-based DG in active distribution networks11The short version of the paper was presented at CUE2023. This paper is a substantial extension of the short version of the conference paper," Applied Energy, Elsevier, vol. 372(C).
    4. Jude Suchithra & Amin Rajabi & Duane A. Robinson, 2024. "Enhancing PV Hosting Capacity of Electricity Distribution Networks Using Deep Reinforcement Learning-Based Coordinated Voltage Control," Energies, MDPI, vol. 17(20), pages 1-27, October.
    5. Jude Suchithra & Duane Robinson & Amin Rajabi, 2023. "Hosting Capacity Assessment Strategies and Reinforcement Learning Methods for Coordinated Voltage Control in Electricity Distribution Networks: A Review," Energies, MDPI, vol. 16(5), pages 1-28, March.
    6. Yu, Nanpeng & Zhang, Shaorong & Qin, Jingtao & Hidalgo-Gonzalez, Patricia & Dobbe, Roel & Liu, Yang & Dubey, Anamika & Wang, Yubo & Dirkman, John & Zhong, Haiwang & Lu, Ning & Ma, Emily & Ding, Zhaoha, 2025. "Data-driven control, optimization, and decision-making in active power distribution networks," Applied Energy, Elsevier, vol. 397(C).
    7. Wu, Zhi & Li, Yiqi & Zhang, Xiao & Zheng, Shu & Zhao, Jingtao, 2025. "Distributed voltage control for multi-feeder distribution networks considering transmission network voltage fluctuation based on robust deep reinforcement learning," Applied Energy, Elsevier, vol. 379(C).
    8. Li, Min & Yang, Yi & He, Zhaoshuang & Guo, Xinbo & Zhang, Ruisheng & Huang, Bingqing, 2023. "A wind speed forecasting model based on multi-objective algorithm and interpretability learning," Energy, Elsevier, vol. 269(C).
    9. Yu, Peipei & Zhang, Hongcai & Song, Yonghua & Wang, Zhenyi & Dong, Huiyu & Ji, Liang, 2025. "Safe reinforcement learning for power system control: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 223(C).
    10. Liu, Xiaolei & Lin, Zi & Feng, Ziming, 2021. "Short-term offshore wind speed forecast by seasonal ARIMA - A comparison against GRU and LSTM," Energy, Elsevier, vol. 227(C).
    11. Lim, Juin Yau & Safder, Usman & How, Bing Shen & Ifaei, Pouya & Yoo, Chang Kyoo, 2021. "Nationwide sustainable renewable energy and Power-to-X deployment planning in South Korea assisted with forecasting model," Applied Energy, Elsevier, vol. 283(C).
    12. Liang, Tao & Zhao, Qing & Lv, Qingzhao & Sun, Hexu, 2021. "A novel wind speed prediction strategy based on Bi-LSTM, MOOFADA and transfer learning for centralized control centers," Energy, Elsevier, vol. 230(C).
    13. Wang, Ying & Li, Hongmin & Jahanger, Atif & Li, Qiwei & Wang, Biao & Balsalobre-Lorente, Daniel, 2024. "A novel ensemble electricity load forecasting system based on a decomposition-selection-optimization strategy," Energy, Elsevier, vol. 312(C).
    14. Liu, Xingdou & Zhang, Li & Wang, Jiangong & Zhou, Yue & Gan, Wei, 2023. "A unified multi-step wind speed forecasting framework based on numerical weather prediction grids and wind farm monitoring data," Renewable Energy, Elsevier, vol. 211(C), pages 948-963.
    15. Tian, Zhongda & Chen, Hao, 2021. "Multi-step short-term wind speed prediction based on integrated multi-model fusion," Applied Energy, Elsevier, vol. 298(C).
    16. Yao, Xianshuang & Guo, Kangshuai & Lei, Jianqi & Li, Xuanyu, 2024. "Fully connected multi-reservoir echo state networks for wind power prediction," Energy, Elsevier, vol. 312(C).
    17. Zhipeng Jing & Lipo Gao & Chengao Wu & Dong Liang, 2025. "Linear Quadratic Regulator-Based Coordinated Voltage and Power Control for Flexible Distribution Networks," Energies, MDPI, vol. 18(2), pages 1-16, January.
    18. Rathore, Abhijeet & Gupta, Priya & Sharma, Raksha & Singh, Rhythm, 2025. "Day ahead solar forecast using long short term memory network augmented with Fast Fourier transform-assisted decomposition technique," Renewable Energy, Elsevier, vol. 247(C).
    19. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    20. Zhu, Xingxu & Hou, Xiangchen & Li, Junhui & Yan, Gangui & Li, Cuiping & Wang, Dongbo, 2023. "Distributed online prediction optimization algorithm for distributed energy resources considering the multi-periods optimal operation," Applied Energy, Elsevier, vol. 348(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:349:y:2023:i:c:s0306261923010127. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.