IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v17y2025i2p407-d1562054.html
   My bibliography  Save this article

Deep Reinforcement Learning-Based Real-Time Energy Management for an Integrated Electric–Thermal Energy System

Author

Listed:
  • Qiang Shuai

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Yue Yin

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Shan Huang

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Chao Chen

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

Abstract

Renewable energy plays a crucial role in achieving sustainable development and has the potential to meet humanity’s long-term energy requirements. Integrated electric–thermal energy systems are an important way to consume a high proportion of renewable energy. The intermittency and volatility of integrated electric–thermal energy systems make solving energy management optimization problems difficult. Thus, this paper proposes an energy management optimization method for an integrated electric–thermal energy system based on the improved proximal policy optimization algorithm, which effectively mitigates the problems of the traditional heuristic algorithms or mathematical planning methods with low accuracy and low solving efficiency. Meanwhile, the proposed algorithm enhances both the convergence speed and overall performance compared to the proximal policy optimization algorithm. This paper first establishes a mathematical model for the energy management of an integrated electric–thermal energy system. Then, the model is formulated as a Markov decision process, and a reward mechanism is designed to guide the agent to learn the uncertainty characteristics of renewable energy output and load consumption in the system through historical data. Finally, in the case study section, the proposed algorithm reduces the average running cost by 2.32% compared to the other algorithms discussed in this paper, thereby demonstrating its effectiveness and cost-efficiency.

Suggested Citation

  • Qiang Shuai & Yue Yin & Shan Huang & Chao Chen, 2025. "Deep Reinforcement Learning-Based Real-Time Energy Management for an Integrated Electric–Thermal Energy System," Sustainability, MDPI, vol. 17(2), pages 1-17, January.
    • Handle: RePEc:gam:jsusta:v:17:y:2025:i:2:p:407-:d:1562054
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/17/2/407/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2071-1050/17/2/407/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yang, Jun & Su, Changqi, 2021. "Robust optimization of microgrid based on renewable distributed power generation and load demand uncertainty," Energy, Elsevier, vol. 223(C).
    2. Wei, Hongqian & Zhang, Youtong & Wang, Yongzhen & Hua, Weiqi & Jing, Rui & Zhou, Yue, 2022. "Planning integrated energy systems coupling V2G as a flexible storage," Energy, Elsevier, vol. 239(PB).
    3. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    4. Jianhong Hao & Ting Huang & Qiuming Xu & Yi Sun, 2023. "Robust Optimal Scheduling of Microgrid with Electric Vehicles Based on Stackelberg Game," Sustainability, MDPI, vol. 15(24), pages 1-15, December.
    5. Huang, Yujing & Wang, Yudong & Liu, Nian, 2022. "A two-stage energy management for heat-electricity integrated energy system considering dynamic pricing of Stackelberg game and operation strategy optimization," Energy, Elsevier, vol. 244(PA).
    6. Chen, Maozhi & Lu, Hao & Chang, Xiqiang & Liao, Haiyan, 2023. "An optimization on an integrated energy system of combined heat and power, carbon capture system and power to gas by considering flexible load," Energy, Elsevier, vol. 273(C).
    7. Ding, Bing & Li, Zening & Li, Zhengmao & Xue, Yixun & Chang, Xinyue & Su, Jia & Jin, Xiaolong & Sun, Hongbin, 2024. "A CCP-based distributed cooperative operation strategy for multi-agent energy systems integrated with wind, solar, and buildings," Applied Energy, Elsevier, vol. 365(C).
    8. Dey, Bishwajit & Misra, Srikant & Garcia Marquez, Fausto Pedro, 2023. "Microgrid system energy management with demand response program for clean and economical operation," Applied Energy, Elsevier, vol. 334(C).
    9. Yin, Yue & Liu, Tianqi & He, Chuan, 2019. "Day-ahead stochastic coordinated scheduling for thermal-hydro-wind-photovoltaic systems," Energy, Elsevier, vol. 187(C).
    10. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gao, Chong & Lin, Junjie & Zeng, Jianfeng & Han, Fengwu, 2022. "Wind-photovoltaic co-generation prediction and energy scheduling of low-carbon complex regional integrated energy system with hydrogen industry chain based on copula-MILP," Applied Energy, Elsevier, vol. 328(C).
    2. Zhao, Wanbing & Chang, Weiguang & Yang, Qiang, 2024. "Collaborative energy management of interconnected regional integrated energy systems considering spatio-temporal characteristics," Renewable Energy, Elsevier, vol. 235(C).
    3. Xu, Xun & Shao, Zhenguo & Chen, Feixiong & Cheng, Guoyang, 2024. "Multi-game optimization operation strategy for integrated energy system considering spatiotemporal correlation of renewable energy," Energy, Elsevier, vol. 303(C).
    4. Yang, Jingxian & Liu, Junyong & Qiu, Gao & Liu, Jichun & Jawad, Shafqat & Zhang, Shuai, 2023. "A spatio-temporality-enabled parallel multi-agent-based real-time dynamic dispatch for hydro-PV-PHS integrated power system," Energy, Elsevier, vol. 278(PB).
    5. Hu, Zhuo & Wang, Tao & Cao, Yuwei & Yang, Qing, 2024. "Electric vehicle aggregator as demand dispatch resources: Exploring the impact of real-time market performance on day-ahead market," Energy, Elsevier, vol. 308(C).
    6. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    7. Kun Li & Yulong Ying & Xiangyu Yu & Jingchao Li, 2024. "Optimal Scheduling of Electricity and Carbon in Multi-Park Integrated Energy Systems," Energies, MDPI, vol. 17(9), pages 1-30, April.
    8. Duan, Jiandong & Tian, Qinxing & Liu, Fan & Xia, Yerui & Gao, Qi, 2024. "Optimal scheduling strategy with integrated demand response based on stepped incentive mechanism for integrated electricity-gas energy system," Energy, Elsevier, vol. 313(C).
    9. Mohd Bilal & Pitshou N. Bokoro & Gulshan Sharma & Giovanni Pau, 2024. "A Cost-Effective Energy Management Approach for On-Grid Charging of Plug-in Electric Vehicles Integrated with Hybrid Renewable Energy Sources," Energies, MDPI, vol. 17(16), pages 1-35, August.
    10. Zhang, Chengquan & Kitamura, Hiroshi & Goto, Mika, 2024. "Feasibility of vehicle-to-grid (V2G) implementation in Japan: A regional analysis of the electricity supply and demand adjustment market," Energy, Elsevier, vol. 311(C).
    11. Østergaard, P.A. & Lund, H. & Thellufsen, J.Z. & Sorknæs, P. & Mathiesen, B.V., 2022. "Review and validation of EnergyPLAN," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    12. Wang, Jin & Zhao, Zhipeng & Zhou, Jinglin & Cheng, Chuntian & Su, Huaying, 2024. "Co-optimization for day-ahead scheduling and flexibility response mode of a hydro–wind–solar hybrid system considering forecast uncertainty of variable renewable energy," Energy, Elsevier, vol. 311(C).
    13. Zhihan Shi & Weisong Han & Guangming Zhang & Zhiqing Bai & Mingxiang Zhu & Xiaodong Lv, 2022. "Research on Low-Carbon Energy Sharing through the Alliance of Integrated Energy Systems with Multiple Uncertainties," Energies, MDPI, vol. 15(24), pages 1-20, December.
    14. Mostafa Esmaeili Shayan & Gholamhassan Najafi & Barat Ghobadian & Shiva Gorjian & Mohamed Mazlan & Mehdi Samami & Alireza Shabanzadeh, 2022. "Flexible Photovoltaic System on Non-Conventional Surfaces: A Techno-Economic Analysis," Sustainability, MDPI, vol. 14(6), pages 1-14, March.
    15. Ma, Yixiang & Yu, Lean & Zhang, Guoxing & Lu, Zhiming & Wu, Jiaqian, 2023. "Source-load uncertainty-based multi-objective multi-energy complementary optimal scheduling," Renewable Energy, Elsevier, vol. 219(P1).
    16. Sulman Shahzad & Muhammad Abbas Abbasi & Hassan Ali & Muhammad Iqbal & Rania Munir & Heybet Kilic, 2023. "Possibilities, Challenges, and Future Opportunities of Microgrids: A Review," Sustainability, MDPI, vol. 15(8), pages 1-28, April.
    17. Gao, Yang & Ai, Qian & He, Xing & Fan, Songli, 2023. "Coordination for regional integrated energy system through target cascade optimization," Energy, Elsevier, vol. 276(C).
    18. Fernando Andrade & Drielli Peyerl & Claudia A. de Mattos, 2025. "Framework for Investment in Electricity Distribution to Enable Energy Transition," Energies, MDPI, vol. 18(3), pages 1-15, February.
    19. Han, Fengwu & Zeng, Jianfeng & Lin, Junjie & Zhao, Yunlong & Gao, Chong, 2023. "A stochastic hierarchical optimization and revenue allocation approach for multi-regional integrated energy systems based on cooperative games," Applied Energy, Elsevier, vol. 350(C).
    20. Alain Aoun & Mehdi Adda & Adrian Ilinca & Mazen Ghandour & Hussein Ibrahim, 2024. "Optimizing Virtual Power Plant Management: A Novel MILP Algorithm to Minimize Levelized Cost of Energy, Technical Losses, and Greenhouse Gas Emissions," Energies, MDPI, vol. 17(16), pages 1-23, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:17:y:2025:i:2:p:407-:d:1562054. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.