IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v313y2022ics0306261922002148.html
   My bibliography  Save this article

Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks

Author

Listed:
  • Gao, Yuanqi
  • Yu, Nanpeng

Abstract

Volt-VAR control (VVC) is a critical tool to manage voltage profiles and reactive power flow in power distribution networks by setting voltage regulating and reactive power compensation device status. To facilitate the adoption of VVC, many physical model-based and data-driven algorithms have been proposed. However, most of the physical model-based methods rely on distribution network parameters, whereas the data-driven algorithms lack safety guarantees. In this paper, we propose a data-driven safe reinforcement learning (RL) algorithm for the VVC problem. We introduce three innovations to improve the learning efficiency and the safety. First, we train the RL agent using a learned environment model to improve the sample efficiency. Second, a safety layer is added to the policy neural network to enhance operational constraint satisfactions for both initial exploration phase and convergence phase. Finally, to improve the algorithm’s performance when learning from limited data, we propose a novel mutual information regularization neural network for the safety layer. Simulation results on IEEE distribution test feeders show that the proposed algorithm improves constraint satisfactions compared to existing data-driven RL methods. With a modest amount of historical data, it is able to approximately maintain constraint satisfactions during the entire course of training. Asymptotically, it also yields similar level of performance of an ideal physical model-based benchmark. One possible limitation is that the proposed framework assumes a time-invariant distribution network topology and zero load transfer from other circuits. This is also an opportunity for future research.

Suggested Citation

  • Gao, Yuanqi & Yu, Nanpeng, 2022. "Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks," Applied Energy, Elsevier, vol. 313(C).
  • Handle: RePEc:eee:appene:v:313:y:2022:i:c:s0306261922002148
    DOI: 10.1016/j.apenergy.2022.118762
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922002148
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2022.118762?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhang, Zhengfa & da Silva, Filipe Faria & Guo, Yifei & Bak, Claus Leth & Chen, Zhe, 2021. "Double-layer stochastic model predictive voltage control in active distribution networks with high penetration of renewables," Applied Energy, Elsevier, vol. 302(C).
    2. Cao, Di & Zhao, Junbo & Hu, Weihao & Ding, Fei & Yu, Nanpeng & Huang, Qi & Chen, Zhe, 2022. "Model-free voltage control of active distribution system with PVs using surrogate model-based deep reinforcement learning," Applied Energy, Elsevier, vol. 306(PA).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Guo, Guodong & Zhang, Mengfan & Gong, Yanfeng & Xu, Qianwen, 2023. "Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay," Applied Energy, Elsevier, vol. 349(C).
    2. Kabir, Farzana & Yu, Nanpeng & Gao, Yuanqi & Wang, Wenyu, 2023. "Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems," Applied Energy, Elsevier, vol. 335(C).
    3. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    4. Alex Chamba & Carlos Barrera-Singaña & Hugo Arcos, 2023. "Optimal Reactive Power Dispatch in Electric Transmission Systems Using the Multi-Agent Model with Volt-VAR Control," Energies, MDPI, vol. 16(13), pages 1-25, June.
    5. Chen, Yongdong & Liu, Youbo & Zhao, Junbo & Qiu, Gao & Yin, Hang & Li, Zhengbo, 2023. "Physical-assisted multi-agent graph reinforcement learning enabled fast voltage regulation for PV-rich active distribution network," Applied Energy, Elsevier, vol. 351(C).
    6. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhu, Xingxu & Hou, Xiangchen & Li, Junhui & Yan, Gangui & Li, Cuiping & Wang, Dongbo, 2023. "Distributed online prediction optimization algorithm for distributed energy resources considering the multi-periods optimal operation," Applied Energy, Elsevier, vol. 348(C).
    2. Zhang, Bin & Hu, Weihao & Ghias, Amer M.Y.M. & Xu, Xiao & Chen, Zhe, 2022. "Multi-agent deep reinforcement learning-based coordination control for grid-aware multi-buildings," Applied Energy, Elsevier, vol. 328(C).
    3. Yin, Linfei & He, Xiaoyu, 2023. "Artificial emotional deep Q learning for real-time smart voltage control of cyber-physical social power systems," Energy, Elsevier, vol. 273(C).
    4. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    5. Xiong, Kang & Hu, Weihao & Cao, Di & Li, Sichen & Zhang, Guozhou & Liu, Wen & Huang, Qi & Chen, Zhe, 2023. "Coordinated energy management strategy for multi-energy hub with thermo-electrochemical effect based power-to-ammonia: A multi-agent deep reinforcement learning enabled approach," Renewable Energy, Elsevier, vol. 214(C), pages 216-232.
    6. Nayak, Dhyaan Sandeep & Misra, Shamik, 2024. "An operational scheduling framework for Electric Vehicle Battery Swapping Station under demand uncertainty," Energy, Elsevier, vol. 290(C).
    7. Guo, Guodong & Zhang, Mengfan & Gong, Yanfeng & Xu, Qianwen, 2023. "Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay," Applied Energy, Elsevier, vol. 349(C).
    8. Liang, Weikun & Lin, Shunjiang & Liu, Mingbo & Sheng, Xuan & Pan, Yue, 2024. "Risk-based distributionally robust optimal dispatch for multiple cascading failures in regional integrated energy system using surrogate modeling," Applied Energy, Elsevier, vol. 353(PA).
    9. Yuan, Yi & Ding, Tao & Chang, Xinyue & Jia, Wenhao & Xue, Yixun, 2024. "A distributed multi-objective optimization method for scheduling of integrated electricity and hydrogen systems," Applied Energy, Elsevier, vol. 355(C).
    10. Jude Suchithra & Duane Robinson & Amin Rajabi, 2023. "Hosting Capacity Assessment Strategies and Reinforcement Learning Methods for Coordinated Voltage Control in Electricity Distribution Networks: A Review," Energies, MDPI, vol. 16(5), pages 1-28, March.
    11. Lin, Hanyang & Shen, Xinwei & Guo, Ye & Ding, Tao & Sun, Hongbin, 2024. "A linear Distflow model considering line shunts for fast calculation and voltage control of power distribution systems," Applied Energy, Elsevier, vol. 357(C).
    12. Qingyan Li & Tao Lin & Qianyi Yu & Hui Du & Jun Li & Xiyue Fu, 2023. "Review of Deep Reinforcement Learning and Its Application in Modern Renewable Power System Control," Energies, MDPI, vol. 16(10), pages 1-23, May.
    13. Zhang, Bin & Hu, Weihao & Xu, Xiao & Li, Tao & Zhang, Zhenyuan & Chen, Zhe, 2022. "Physical-model-free intelligent energy management for a grid-connected hybrid wind-microturbine-PV-EV energy system via deep reinforcement learning approach," Renewable Energy, Elsevier, vol. 200(C), pages 433-448.
    14. Kang, Wenfa & Chen, Minyou & Guan, Yajuan & Wei, Baoze & Vasquez Q., Juan C. & Guerrero, Josep M., 2022. "Event-triggered distributed voltage regulation by heterogeneous BESS in low-voltage distribution networks," Applied Energy, Elsevier, vol. 312(C).
    15. Utama, Christian & Meske, Christian & Schneider, Johannes & Ulbrich, Carolin, 2022. "Reactive power control in photovoltaic systems through (explainable) artificial intelligence," Applied Energy, Elsevier, vol. 328(C).
    16. Yuan, Quan & Ye, Yujian & Tang, Yi & Liu, Yuanchang & Strbac, Goran, 2022. "A novel deep-learning based surrogate modeling of stochastic electric vehicle traffic user equilibrium in low-carbon electricity–transportation nexus," Applied Energy, Elsevier, vol. 315(C).
    17. Jude Suchithra & Duane A. Robinson & Amin Rajabi, 2024. "A Model-Free Deep Reinforcement Learning-Based Approach for Assessment of Real-Time PV Hosting Capacity," Energies, MDPI, vol. 17(9), pages 1-12, April.
    18. Zhang, Xiao & Wu, Zhi & Sun, Qirun & Gu, Wei & Zheng, Shu & Zhao, Jingtao, 2024. "Application and progress of artificial intelligence technology in the field of distribution network voltage Control:A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:313:y:2022:i:c:s0306261922002148. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.