IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v18y2025i5p1020-d1595435.html
   My bibliography  Save this article

Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks

Author

Listed:
  • Gang Lu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Bo Yuan

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Baorui Nie

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Peng Xia

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Cong Wu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Guangzeng Sun

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

Abstract

The increasing integration of renewable energy-based distributed generation (DG) in modern distribution networks is essential for reducing reliance on fossil fuels. However, the unpredictability and intermittency of renewable sources such as wind and photovoltaic (PV) systems introduce significant challenges for distribution network planning. To address these challenges, this paper proposes a Q-learning-based Distributionally Robust Optimization (DRO) model for expansion planning of distribution networks and generation units. The proposed model incorporates energy storage systems (ESSs), renewable DG, substations, and distribution lines while considering uncertainties such as renewable generation variability, load fluctuations, and system contingencies. Through a dynamic decision-making process using Q-learning, the model adapts to changing network conditions to minimize the total system cost while maintaining reliability. The Latin Hypercube Sampling (LHS) method is employed to generate multi-scenario data, and piecewise linearization is used to reduce the computational complexity of the AC power flow equations. Numerical results demonstrate that the model significantly improves system reliability and economic efficiency under multiple uncertainty scenarios. The results also highlight the crucial role of the ESS in mitigating the variability of renewable energy and reducing the expected energy not supplied (EENS).

Suggested Citation

  • Gang Lu & Bo Yuan & Baorui Nie & Peng Xia & Cong Wu & Guangzeng Sun, 2025. "Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks," Energies, MDPI, vol. 18(5), pages 1-25, February.
    • Handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/18/5/1020/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/18/5/1020/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Canizes, Bruno & Soares, João & Lezama, Fernando & Silva, Cátia & Vale, Zita & Corchado, Juan M., 2019. "Optimal expansion planning considering storage investment and seasonal effect of demand and renewable generation," Renewable Energy, Elsevier, vol. 138(C), pages 937-954.
    2. Li, Rui & Wang, Wei & Wu, Xuezhi & Tang, Fen & Chen, Zhe, 2019. "Cooperative planning model of renewable energy sources and energy storage units in active distribution systems: A bi-level model and Pareto analysis," Energy, Elsevier, vol. 168(C), pages 30-42.
    3. Yang, Zhixue & Ren, Zhouyang & Li, Hui & Sun, Zhiyuan & Feng, Jianbing & Xia, Weiyi, 2024. "A multi-stage stochastic dispatching method for electricity‑hydrogen integrated energy systems driven by model and data," Applied Energy, Elsevier, vol. 371(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhou, Siyu & Han, Yang & Chen, Shuheng & Yang, Ping & Mahmoud, Karar & Darwish, Mohamed M.F. & Matti, Lehtonen & Zalhaf, Amr S., 2023. "A multiple uncertainty-based Bi-level expansion planning paradigm for distribution networks complying with energy storage system functionalities," Energy, Elsevier, vol. 275(C).
    2. Wang, Bangyan & Wang, Xiuli & Zhu, Zongyao & Wu, Xiong, 2023. "Siting and sizing of energy storage for renewable generation utilization with multi-stage dispatch under uncertainty: A tri-level model and decomposition approach," Applied Energy, Elsevier, vol. 344(C).
    3. Li, Jinghua & Lu, Bo & Wang, Zhibang & Zhu, Mengshu, 2021. "Bi-level optimal planning model for energy storage systems in a virtual power plant," Renewable Energy, Elsevier, vol. 165(P2), pages 77-95.
    4. de Lima, Tayenne Dias & Lezama, Fernando & Soares, João & Franco, John F. & Vale, Zita, 2024. "Modern distribution system expansion planning considering new market designs: Review and future directions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    5. Ghahramani, Mehrdad & Nazari-Heris, Morteza & Zare, Kazem & Mohammadi-Ivatloo, Behnam, 2019. "Energy and reserve management of a smart distribution system by incorporating responsive-loads /battery/wind turbines considering uncertain parameters," Energy, Elsevier, vol. 183(C), pages 205-219.
    6. Liu, Sha & Shen, Jiong & Zhang, Junli, 2024. "A novel configuration optimization approach for IES considering exergy-degradation and non-energy costs of equipment," Energy, Elsevier, vol. 312(C).
    7. Xi Zhang & Longyun Kang & Xuemei Wang & Yangbo Liu & Sheng Huang, 2025. "Capacity Optimization Configuration of Hybrid Energy Storage Systems for Wind Farms Based on Improved k-means and Two-Stage Decomposition," Energies, MDPI, vol. 18(4), pages 1-24, February.
    8. Mohammed, Nooriya A. & Al-Bazi, Ammar, 2021. "Management of renewable energy production and distribution planning using agent-based modelling," Renewable Energy, Elsevier, vol. 164(C), pages 509-520.
    9. Mateusz Andrychowicz, 2021. "RES and ES Integration in Combination with Distribution Grid Development Using MILP," Energies, MDPI, vol. 14(2), pages 1-19, January.
    10. Wang, Ying & Chen, Ying & Xiao, Xianyong & Chen, Yunzhu & Li, Qilin, 2025. "Bi-level planning-operation model of PV considering reactive power capability," Applied Energy, Elsevier, vol. 378(PA).
    11. Qiao, Jinpeng & Mi, Yang & Shen, Jie & Lu, Changkun & Cai, Pengcheng & Ma, Siyuan & Wang, Peng, 2025. "Optimization schedule strategy of active distribution network based on microgrid group and shared energy storage," Applied Energy, Elsevier, vol. 377(PD).
    12. Huang, Congzhi & Sheng, Xinxin, 2020. "Data-driven model identification of boiler-turbine coupled process in 1000 MW ultra-supercritical unit by improved bird swarm algorithm," Energy, Elsevier, vol. 205(C).
    13. Yamchi, Hamid Bakhshi & Safari, Amin & Guerrero, Josep M., 2021. "A multi-objective mixed integer linear programming model for integrated electricity-gas network expansion planning considering the impact of photovoltaic generation," Energy, Elsevier, vol. 222(C).
    14. Zhang, Jingrui & Zhu, Xiaoqing & Chen, Tengpeng & Yu, Yanlin & Xue, Wendong, 2020. "Improved MOEA/D approach to many-objective day-ahead scheduling with consideration of adjustable outputs of renewable units and load reduction in active distribution networks," Energy, Elsevier, vol. 210(C).
    15. Gu, Chenjia & Wang, Jianxue & Zhang, Yao & Li, Qingtao & Chen, Yang, 2022. "Optimal energy storage planning for stacked benefits in power distribution network," Renewable Energy, Elsevier, vol. 195(C), pages 366-380.
    16. Arteaga, Juan & Zareipour, Hamidreza & Amjady, Nima, 2021. "Energy Storage as a Service: Optimal sizing for Transmission Congestion Relief," Applied Energy, Elsevier, vol. 298(C).
    17. Rastgou, Abdollah, 2024. "Distribution network expansion planning: An updated review of current methods and new challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    18. Yao Xiao & Caixia Yang & Tao Chen & Mingze Lei & Supannika Wattana & Buncha Wattana, 2025. "Strategies of a Wind–Solar–Storage System in Jiangxi Province Using the LEAP–NEMO Framework for Achieving Carbon Peaking Goals," Energies, MDPI, vol. 18(5), pages 1-19, February.
    19. Xie, Shiwei & Hu, Zhijian & Wang, Jueying, 2020. "Two-stage robust optimization for expansion planning of active distribution systems coupled with urban transportation networks," Applied Energy, Elsevier, vol. 261(C).
    20. Li, Xiangrong & Zhu, Shaoying & Yüksel, Serhat & Dinçer, Hasan & Ubay, Gözde Gülseven, 2020. "Kano-based mapping of innovation strategies for renewable energy alternatives using hybrid interval type-2 fuzzy decision-making approach," Energy, Elsevier, vol. 211(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.