IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v376y2024ipas0306261924013680.html
   My bibliography  Save this article

Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method

Author

Listed:
  • Xu, Biao
  • Luan, Wenpeng
  • Yang, Jing
  • Zhao, Bochao
  • Long, Chao
  • Ai, Qian
  • Xiang, Jiani

Abstract

Virtual power plant (VPP) emerges as a promising integration and aggregation technology that facilitates the utilization of massive flexible demand-side resources (DSRs). However, non-negligible modeling errors and high-dimensional uncertainties involved in DSR aggregation threaten the delivery reliability and cost-effectiveness of VPP operation. To address this problem, this study proposes an integrated three-stage scheduling framework for VPPs and develops a model-assisted multi-agent reinforcement learning (MARL) approach. In the proposed framework, the VPP scheduling problem is formulated as a decentralized partially observable Markov Decision Process (Dec-POMDP), which depicts the complex interaction process among the three stages (bidding, re-dispatching and disaggregation). The interactions are evaluated by a comprehensive reward function, incorporating the trading and operation costs, as well as imbalance penalties. To enable decentralized decision-making, a model-assisted multi-agent proximal policy optimization (MA2PPO) algorithm is proposed, which trains a separate actor network for each aggregator. Additionally, the MA2PPO is augmented with a model-assisted safety decision-making method to accelerate the training process. Numerical simulation results verify that the proposed method enhances the delivery reliability and cost-effectiveness of the VPP, while achieving faster convergence time compared with purely model-free MARL methods.

Suggested Citation

  • Xu, Biao & Luan, Wenpeng & Yang, Jing & Zhao, Bochao & Long, Chao & Ai, Qian & Xiang, Jiani, 2024. "Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method," Applied Energy, Elsevier, vol. 376(PA).
    • Handle: RePEc:eee:appene:v:376:y:2024:i:pa:s0306261924013680
    DOI: 10.1016/j.apenergy.2024.123985
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924013680
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123985?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Hasankhani, Arezoo & Hakimi, Seyed Mehdi, 2021. "Stochastic energy management of smart microgrid with intermittent renewable energy resources in electricity market," Energy, Elsevier, vol. 219(C).
    2. Shayegan-Rad, Ali & Badri, Ali & Zangeneh, Ali, 2017. "Day-ahead scheduling of virtual power plant in joint energy and regulation reserve markets under uncertainties," Energy, Elsevier, vol. 121(C), pages 114-125.
    3. Ding, Yixing & Xu, Qingshan & Hao, Lili & Xia, Yuanxing, 2023. "A Stackelberg Game-based robust optimization for user-side energy storage configuration and power pricing," Energy, Elsevier, vol. 283(C).
    4. Li, Xiangyu & Luo, Fengji & Li, Chaojie, 2024. "Multi-agent deep reinforcement learning-based autonomous decision-making framework for community virtual power plants," Applied Energy, Elsevier, vol. 360(C).
    5. Li, Xinchao & Lu, Shan & Li, Zhe & Wang, Yue & Zhu, Li, 2022. "Modeling and optimization of bioethanol production planning under hybrid uncertainty: A heuristic multi-stage stochastic programming approach," Energy, Elsevier, vol. 245(C).
    6. Iria, José & Soares, Filipe, 2019. "Real-time provision of multiple electricity market products by an aggregator of prosumers," Applied Energy, Elsevier, vol. 255(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xinxing Liu & Ciwei Gao, 2025. "Review and Prospects of Artificial Intelligence Technology in Virtual Power Plants," Energies, MDPI, vol. 18(13), pages 1-26, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Caixin Yan & Zhifeng Qiu, 2025. "Review of Power Market Optimization Strategies Based on Industrial Load Flexibility," Energies, MDPI, vol. 18(7), pages 1-41, March.
    2. Seong-Hyeon Cha & Sun-Hyeok Kwak & Woong Ko, 2023. "A Robust Optimization Model of Aggregated Resources Considering Serving Ratio for Providing Reserve Power in the Joint Electricity Market," Energies, MDPI, vol. 16(20), pages 1-27, October.
    3. Wang, Tonghe & Hua, Haochen & Shi, Tianying & Wang, Rui & Sun, Yizhong & Naidoo, Pathmanathan, 2024. "A bi-level dispatch optimization of multi-microgrid considering green electricity consumption willingness under renewable portfolio standard policy," Applied Energy, Elsevier, vol. 356(C).
    4. Hanwen Wang & Xiang Li & Haojun Hu & Yizhou Zhou, 2024. "Distributed Dispatch and Profit Allocation for Parks Using Co-Operative Game Theory and the Generalized Nash Bargaining Approach," Energies, MDPI, vol. 17(23), pages 1-19, December.
    5. Jani, Ali & Karimi, Hamid & Jadid, Shahram, 2022. "Two-layer stochastic day-ahead and real-time energy management of networked microgrids considering integration of renewable energy resources," Applied Energy, Elsevier, vol. 323(C).
    6. Wang, Bangyan & Wang, Xiuli & Wei, Fengting & Shao, Chengcheng & Zhou, Jiahao & Lin, Jintian, 2023. "Multi-stage stochastic planning for a long-term low-carbon transition of island power system considering carbon price uncertainty and offshore wind power," Energy, Elsevier, vol. 282(C).
    7. Xuejun Zheng & Shaorong Wang & Xin Su & Mengmeng Xiao & Zia Ullah & Xin Hu & Chang Ye, 2021. "Real-Time Dynamic Behavior Evaluation of Active Distribution Networks Leveraging Low-Cost PMUs," Energies, MDPI, vol. 14(16), pages 1-20, August.
    8. Magda I. El-Afifi & Magdi M. Saadawi & Abdelfattah A. Eladl, 2022. "Cogeneration Systems Performance Analysis as a Sustainable Clean Energy and Water Source Based on Energy Hubs Using the Archimedes Optimization Algorithm," Sustainability, MDPI, vol. 14(22), pages 1-26, November.
    9. Li, Xudong & Yang, Weijia & Liao, Yiwen & Zhang, Shushu & Zheng, Yang & Zhao, Zhigao & Tang, Maojia & Cheng, Yongguang & Liu, Pan, 2024. "Short-term risk-management for hydro-wind-solar hybrid energy system considering hydropower part-load operating characteristics," Applied Energy, Elsevier, vol. 360(C).
    10. Cao, K.H. & Qi, H.S. & Tsai, C.H. & Woo, C.K. & Zarnikau, J., 2021. "Energy trading efficiency in the US Midcontinent electricity markets," Applied Energy, Elsevier, vol. 302(C).
    11. Kaiyan Wang & Xueyan Wang & Rong Jia & Jian Dang & Yan Liang & Haodong Du, 2022. "Research on Coupled Cooperative Operation of Medium- and Long-Term and Spot Electricity Transaction for Multi-Energy System: A Case Study in China," Sustainability, MDPI, vol. 14(17), pages 1-20, August.
    12. Rui Gao & Hongxia Guo & Ruihong Zhang & Tian Mao & Qianyao Xu & Baorong Zhou & Ping Yang, 2019. "A Two-Stage Dispatch Mechanism for Virtual Power Plant Utilizing the CVaR Theory in the Electricity Spot Market," Energies, MDPI, vol. 12(17), pages 1-18, September.
    13. Li, Jinchao & Sun, Zihao & Niu, Xiaoxuan & Li, Shiwei, 2024. "Economic optimization scheduling of virtual power plants considering an incentive based tiered carbon price," Energy, Elsevier, vol. 305(C).
    14. Elkholy, M.H. & Metwally, Hamid & Farahat, M.A. & Senjyu, Tomonobu & Elsayed Lotfy, Mohammed, 2022. "Smart centralized energy management system for autonomous microgrid using FPGA," Applied Energy, Elsevier, vol. 317(C).
    15. Yin, Linfei & Qiu, Yao, 2022. "Long-term price guidance mechanism of flexible energy service providers based on stochastic differential methods," Energy, Elsevier, vol. 238(PB).
    16. Nemanja Mišljenović & Matej Žnidarec & Goran Knežević & Damir Šljivac & Andreas Sumper, 2023. "A Review of Energy Management Systems and Organizational Structures of Prosumers," Energies, MDPI, vol. 16(7), pages 1-32, March.
    17. Liu, Shuhan & Sun, Wenqiang, 2025. "Knowledge- and data-driven prediction of blast furnace gas generation and consumption in iron and steel sites," Applied Energy, Elsevier, vol. 390(C).
    18. Srđan Skok & Ahmed Mutapčić & Renata Rubesa & Mario Bazina, 2020. "Transmission Power System Modeling by Using Aggregated Distributed Generation Model Based on a TSO—DSO Data Exchange Scheme," Energies, MDPI, vol. 13(15), pages 1-15, August.
    19. Guo, Xiaopeng & Wang, Liyi & Ren, Dongfang, 2025. "Optimal scheduling model for virtual power plant combining carbon trading and green certificate trading," Energy, Elsevier, vol. 318(C).
    20. Chang, Yunwei & Gu, Heng & Yao, Xiaoyan & Qing, Chunyao & Zou, Deqiu, 2024. "Preparation of a novel microencapsulated phase change material (MEPCM)/adipic acid ceramic composite and its thermal performance," Energy, Elsevier, vol. 292(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:376:y:2024:i:pa:s0306261924013680. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.