IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v18y2025i5p1020-d1595435.html
   My bibliography  Save this article

Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks

Author

Listed:
  • Gang Lu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Bo Yuan

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Baorui Nie

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Peng Xia

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Cong Wu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Guangzeng Sun

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

Abstract

The increasing integration of renewable energy-based distributed generation (DG) in modern distribution networks is essential for reducing reliance on fossil fuels. However, the unpredictability and intermittency of renewable sources such as wind and photovoltaic (PV) systems introduce significant challenges for distribution network planning. To address these challenges, this paper proposes a Q-learning-based Distributionally Robust Optimization (DRO) model for expansion planning of distribution networks and generation units. The proposed model incorporates energy storage systems (ESSs), renewable DG, substations, and distribution lines while considering uncertainties such as renewable generation variability, load fluctuations, and system contingencies. Through a dynamic decision-making process using Q-learning, the model adapts to changing network conditions to minimize the total system cost while maintaining reliability. The Latin Hypercube Sampling (LHS) method is employed to generate multi-scenario data, and piecewise linearization is used to reduce the computational complexity of the AC power flow equations. Numerical results demonstrate that the model significantly improves system reliability and economic efficiency under multiple uncertainty scenarios. The results also highlight the crucial role of the ESS in mitigating the variability of renewable energy and reducing the expected energy not supplied (EENS).

Suggested Citation

  • Gang Lu & Bo Yuan & Baorui Nie & Peng Xia & Cong Wu & Guangzeng Sun, 2025. "Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks," Energies, MDPI, vol. 18(5), pages 1-25, February.
    • Handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/18/5/1020/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/18/5/1020/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Canizes, Bruno & Soares, João & Lezama, Fernando & Silva, Cátia & Vale, Zita & Corchado, Juan M., 2019. "Optimal expansion planning considering storage investment and seasonal effect of demand and renewable generation," Renewable Energy, Elsevier, vol. 138(C), pages 937-954.
    2. Li, Rui & Wang, Wei & Wu, Xuezhi & Tang, Fen & Chen, Zhe, 2019. "Cooperative planning model of renewable energy sources and energy storage units in active distribution systems: A bi-level model and Pareto analysis," Energy, Elsevier, vol. 168(C), pages 30-42.
    3. Li, Xin & Wu, Xian & Gui, De & Hua, Yawen & Guo, Panfeng, 2021. "Power system planning based on CSP-CHP system to integrate variable renewable energy," Energy, Elsevier, vol. 232(C).
    4. Yang, Zhixue & Ren, Zhouyang & Li, Hui & Sun, Zhiyuan & Feng, Jianbing & Xia, Weiyi, 2024. "A multi-stage stochastic dispatching method for electricity‑hydrogen integrated energy systems driven by model and data," Applied Energy, Elsevier, vol. 371(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhou, Siyu & Han, Yang & Chen, Shuheng & Yang, Ping & Mahmoud, Karar & Darwish, Mohamed M.F. & Matti, Lehtonen & Zalhaf, Amr S., 2023. "A multiple uncertainty-based Bi-level expansion planning paradigm for distribution networks complying with energy storage system functionalities," Energy, Elsevier, vol. 275(C).
    2. Wang, Bangyan & Wang, Xiuli & Zhu, Zongyao & Wu, Xiong, 2023. "Siting and sizing of energy storage for renewable generation utilization with multi-stage dispatch under uncertainty: A tri-level model and decomposition approach," Applied Energy, Elsevier, vol. 344(C).
    3. Zhou, Siyu & Han, Yang & Yang, Ping & Mahmoud, Karar & Lehtonen, Matti & Darwish, Mohamed M.F. & Zalhaf, Amr S., 2022. "An optimal network constraint-based joint expansion planning model for modern distribution networks with multi-types intermittent RERs," Renewable Energy, Elsevier, vol. 194(C), pages 137-151.
    4. Li, Jinghua & Lu, Bo & Wang, Zhibang & Zhu, Mengshu, 2021. "Bi-level optimal planning model for energy storage systems in a virtual power plant," Renewable Energy, Elsevier, vol. 165(P2), pages 77-95.
    5. Shen, Yi & Zhai, Junyi & Kang, Zhongjian & Zhao, Bei & Gao, Xianhui & Li, Zhengmao, 2025. "Distributionally robust chance-constrained energy management for island DC microgrid with offshore wind power hydrogen production," Energy, Elsevier, vol. 316(C).
    6. de Lima, Tayenne Dias & Lezama, Fernando & Soares, João & Franco, John F. & Vale, Zita, 2024. "Modern distribution system expansion planning considering new market designs: Review and future directions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    7. Zahra Hussaini & Arash Nemati & Mohammad Mahdi Paydar, 2025. "A multi-period multi-season multi-objective mathematical model for guaranteeing the viability of supply chains under fluctuations: a healthcare closed-loop supply chain application," Annals of Operations Research, Springer, vol. 346(2), pages 1063-1108, March.
    8. David Borge-Diez & Enrique Rosales-Asensio & Emin Açıkkalp & Daniel Alonso-Martínez, 2023. "Analysis of Power to Gas Technologies for Energy Intensive Industries in European Union," Energies, MDPI, vol. 16(1), pages 1-22, January.
    9. Keyong Hu & Qingqing Yang & Lei Lu & Yu Zhang & Shuifa Sun & Ben Wang, 2025. "Two-Stage Distributionally Robust Optimal Scheduling for Integrated Energy Systems Considering Uncertainties in Renewable Generation and Loads," Mathematics, MDPI, vol. 13(9), pages 1-30, April.
    10. Li, Jing & Lu, Tianguang & Yi, Xinning & Hao, Ran & Ai, Qian & Guo, Yu & An, Molin & Wang, Shaorui & He, Xueqian & Li, Yixiao, 2024. "Concentrated solar power for a reliable expansion of energy systems with high renewable penetration considering seasonal balance," Renewable Energy, Elsevier, vol. 226(C).
    11. Ghahramani, Mehrdad & Nazari-Heris, Morteza & Zare, Kazem & Mohammadi-Ivatloo, Behnam, 2019. "Energy and reserve management of a smart distribution system by incorporating responsive-loads /battery/wind turbines considering uncertain parameters," Energy, Elsevier, vol. 183(C), pages 205-219.
    12. Chen, Xiaoyuan & Jiang, Shan & Chen, Yu & Lei, Yi & Zhang, Donghui & Zhang, Mingshun & Gou, Huayu & Shen, Boyang, 2022. "A 10 MW class data center with ultra-dense high-efficiency energy distribution: Design and economic evaluation of superconducting DC busbar networks," Energy, Elsevier, vol. 250(C).
    13. Yu, Yanghao & Du, Ershun & Chen, Zhichao & Su, Yibo & Zhang, Xianfeng & Yang, Hongbin & Wang, Peng & Zhang, Ning, 2022. "Optimal portfolio of a 100% renewable energy generation base supported by concentrating solar power," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    14. Liu, Sha & Shen, Jiong & Zhang, Junli, 2024. "A novel configuration optimization approach for IES considering exergy-degradation and non-energy costs of equipment," Energy, Elsevier, vol. 312(C).
    15. Xi Zhang & Longyun Kang & Xuemei Wang & Yangbo Liu & Sheng Huang, 2025. "Capacity Optimization Configuration of Hybrid Energy Storage Systems for Wind Farms Based on Improved k-means and Two-Stage Decomposition," Energies, MDPI, vol. 18(4), pages 1-24, February.
    16. Mohammed, Nooriya A. & Al-Bazi, Ammar, 2021. "Management of renewable energy production and distribution planning using agent-based modelling," Renewable Energy, Elsevier, vol. 164(C), pages 509-520.
    17. Xie, Rui & Wei, Wei & Li, Mingxuan & Dong, ZhaoYang & Mei, Shengwei, 2023. "Sizing capacities of renewable generation, transmission, and energy storage for low-carbon power systems: A distributionally robust optimization approach," Energy, Elsevier, vol. 263(PA).
    18. Ren, Peng & Chen, Lunshu & Hui, Hongxun, 2024. "Power-controllable variable refrigerant flow system with flexibility value for demand response," Energy, Elsevier, vol. 313(C).
    19. Mateusz Andrychowicz, 2021. "RES and ES Integration in Combination with Distribution Grid Development Using MILP," Energies, MDPI, vol. 14(2), pages 1-19, January.
    20. Wang, Ying & Chen, Ying & Xiao, Xianyong & Chen, Yunzhu & Li, Qilin, 2025. "Bi-level planning-operation model of PV considering reactive power capability," Applied Energy, Elsevier, vol. 378(PA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.