IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v391y2025ics030626192500580x.html
   My bibliography  Save this article

A bi-level optimization strategy for flexible and economic operation of the CHP units based on reinforcement learning and multi-objective MPC

Author

Listed:
  • Zhu, Keyan
  • Zhang, Guangming
  • Zhu, Chen
  • Niu, Yuguang
  • Liu, Jizhen

Abstract

Enhancing the comprehensive performance of the combined heat and power (CHP) units is crucial for accommodating renewable energy and achieving energy conservation. To this end, a bi-level optimization strategy based on reinforcement learning (RL) and multi-objective model predictive control (MOMPC) is proposed to enhance the CHP units flexibility and economic performance. Firstly, a CHP unit model is constructed, and its various parameters are incorporated into the rolling optimization of the MOMPC, serving as the lower-level follower to solve the fundamental control. Secondly, a bi-level optimization strategy integrating the twin delayed deep deterministic policy gradient (TD3) algorithm with MOMPC (TD3-MOMPC) is proposed. The TD3 agent is designated as the upper-level leader. By decomposing the complex flexibility requirements and the optimization control sequence of the CHP unit, tasks are assigned to both the upper-level leader and the lower-level follower for bi-level interactive optimization. Thirdly, with power flexibility, heating quality, and operational economy serving as leader guidance, a multi-criterion optimization reward function is designed for the upper-level. Then, the actions of the upper-level TD3 agent are designed as dynamic weights and time-varying prediction horizons for the rolling optimization of MOMPC, serving as a bridge to connect and guide the bi-level optimization. Finally, to verify the effectiveness of the bi-level optimization strategy, extensive tests on load variation and disturbance rejection were conducted on a 300 MW CHP unit. The results show that the proposed strategy enhances the unit's load flexibility, heating quality, and operational economy.

Suggested Citation

  • Zhu, Keyan & Zhang, Guangming & Zhu, Chen & Niu, Yuguang & Liu, Jizhen, 2025. "A bi-level optimization strategy for flexible and economic operation of the CHP units based on reinforcement learning and multi-objective MPC," Applied Energy, Elsevier, vol. 391(C).
    • Handle: RePEc:eee:appene:v:391:y:2025:i:c:s030626192500580x
    DOI: 10.1016/j.apenergy.2025.125850
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S030626192500580X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125850?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Kong, Xiaobing & Abdelbaky, Mohamed Abdelkarim & Liu, Xiangjie & Lee, Kwang Y., 2023. "Stable feedback linearization-based economic MPC scheme for thermal power plant," Energy, Elsevier, vol. 268(C).
    2. Yan, Rujing & Wang, Jiangjiang & Huo, Shuojie & Zhang, Jing & Tang, Saiqiu & Yang, Mei, 2023. "Comparative study for four technologies on flexibility improvement and renewable energy accommodation of combined heat and power system," Energy, Elsevier, vol. 263(PE).
    3. Hou, Guolian & Gong, Linjuan & Hu, Bo & Su, Huilin & Huang, Ting & Huang, Congzhi & Fan, Wei & Zhao, Yuanzhu, 2022. "Application of fast adaptive moth-flame optimization in flexible operation modeling for supercritical unit," Energy, Elsevier, vol. 239(PA).
    4. Kortela, J. & Jämsä-Jounela, S.-L., 2014. "Model predictive control utilizing fuel and moisture soft-sensors for the BioPower 5 combined heat and power (CHP) plant," Applied Energy, Elsevier, vol. 131(C), pages 189-200.
    5. Wang, Zhu & Liu, Ming & Yan, Hui & Yan, Junjie, 2022. "Optimization on coordinate control strategy assisted by high-pressure extraction steam throttling to achieve flexible and efficient operation of thermal power plants," Energy, Elsevier, vol. 244(PA).
    6. Zhao, Yongliang & Liu, Ming & Wang, Chaoyang & Li, Xin & Chong, Daotong & Yan, Junjie, 2018. "Increasing operational flexibility of supercritical coal-fired power plants by regulating thermal system configuration during transient processes," Applied Energy, Elsevier, vol. 228(C), pages 2375-2386.
    7. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun & Li, Fusheng & Lin, Dan & Zhu, Hanxin, 2021. "Efficient experience replay based deep deterministic policy gradient for AGC dispatch in integrated energy system," Applied Energy, Elsevier, vol. 285(C).
    8. Zhang, Guangming & Zhang, Chao & Wang, Wei & Cao, Huan & Chen, Zhenyu & Niu, Yuguang, 2023. "Offline reinforcement learning control for electricity and heat coordination in a supercritical CHP unit," Energy, Elsevier, vol. 266(C).
    9. Bao, Zhejing & Ye, Yangli & Liu, Ruijie & Cheng, Weidong & Zhao, Qiang & Wu, Ting, 2022. "Scheduling coordination of back pressure CHP coupled electricity-heat energy system with adaptive constraint strategy to accommodate uncertain wind power," Energy, Elsevier, vol. 240(C).
    10. Lv, Chaoxian & Yu, Hao & Li, Peng & Wang, Chengshan & Xu, Xiandong & Li, Shuquan & Wu, Jianzhong, 2019. "Model predictive control based robust scheduling of community integrated energy system with operational flexibility," Applied Energy, Elsevier, vol. 243(C), pages 250-265.
    11. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Huang, Congzhi, 2024. "A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy," Energy, Elsevier, vol. 289(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Huang, Congzhi, 2024. "A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy," Energy, Elsevier, vol. 289(C).
    2. Hou, Guolian & Liu, Zeyu, 2025. "Data-driven modeling for ultra-supercritical unit based on bidirectional test-time training and improved temporal convolutional network," Energy, Elsevier, vol. 326(C).
    3. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Gong, Linjuan & Huang, Congzhi & Zhang, Jianhua, 2023. "Application of multi-agent EADRC in flexible operation of combined heat and power plant considering carbon emission and economy," Energy, Elsevier, vol. 263(PB).
    4. Liu, Zefeng & Wang, Chaoyang & Fan, Jianlin & Liu, Ming & Xing, Yong & Yan, Junjie, 2024. "Enhancing the flexibility and stability of coal-fired power plants by optimizing control schemes of throttling high-pressure extraction steam," Energy, Elsevier, vol. 288(C).
    5. Hou, Guolian & Ye, Lingling & Huang, Ting & Huang, Congzhi, 2024. "Intelligent modeling of combined heat and power unit under full operating conditions via improved crossformer and precise sparrow search algorithm," Energy, Elsevier, vol. 308(C).
    6. Dong, Zhe & Cheng, Zhonghua & Zhu, Yunlong & Huang, Xiaojin & Dong, Yujie & Zhang, Zuoyi, 2023. "Coordinated control of mHTGR-based nuclear steam supply systems considering cold helium temperature," Energy, Elsevier, vol. 284(C).
    7. Du, Zeyu & Liu, Ming & Wang, Yang & Zhou, Yu & Zhao, Yongliang & Yan, Junjie, 2025. "Energy consumption characteristics and energy saving potential of thermal power plants under ultra-low power load ratio conditions," Energy, Elsevier, vol. 330(C).
    8. Chen, Chen & Zhao, Chenyu & Liu, Ming & Wang, Chaoyang & Yan, Junjie, 2024. "Enhancing the load cycling rate of subcritical coal-fired power plants: A novel control strategy based on data-driven feedwater active regulation," Energy, Elsevier, vol. 312(C).
    9. Pang, Dawei & Niu, Yuguang & Du, Ming, 2025. "Phase lead error-based active disturbance rejection control for 1000 MW ultra-supercritical unit under flexible operation," Energy, Elsevier, vol. 319(C).
    10. Hou, Guolian & Huang, Ting & Huang, Congzhi, 2023. "Flexibility improvement of 1000 MW ultra-supercritical unit under full operating conditions by error-based ADRC and fast pigeon-inspired optimizer," Energy, Elsevier, vol. 270(C).
    11. Liu, Xiang & Wu, Fengyongkang & Lv, Laiquan & Wei, Lijia & Zhou, Hao, 2024. "Performance of solid particles as thermal storage media in thermal power flexibility retrofits: Effects of charging and discharging flow rates on single piece stacking bed," Energy, Elsevier, vol. 308(C).
    12. Yin, Linfei & Xie, Jiaxing, 2022. "Multi-feature-scale fusion temporal convolution networks for metal temperature forecasting of ultra-supercritical coal-fired power plant reheater tubes," Energy, Elsevier, vol. 238(PA).
    13. Wang, Zhenpu & Xu, Jing & Ma, Suxia & Zhao, Guanjia & Wang, Jianfei & Gu, Yujiong, 2025. "Comparative investigation on heat pump solutions for peak shaving and heat-power decoupling in combined heat and power plants," Renewable and Sustainable Energy Reviews, Elsevier, vol. 216(C).
    14. Wang, Pengfei & Liang, Wenlong & Gong, Huijun & Chen, Jie, 2024. "Decoupling control of core power and axial power distribution for large pressurized water reactors based on reinforcement learning," Energy, Elsevier, vol. 313(C).
    15. Xu, Jing & Wang, Xiaoying & Gu, Yujiong & Ma, Suxia, 2023. "A data-based day-ahead scheduling optimization approach for regional integrated energy systems with varying operating conditions," Energy, Elsevier, vol. 283(C).
    16. Xiang, Yue & Guo, Yongtao & Wu, Gang & Liu, Junyong & Sun, Wei & Lei, Yutian & Zeng, Pingliang, 2022. "Low-carbon economic planning of integrated electricity-gas energy systems," Energy, Elsevier, vol. 249(C).
    17. Mazare, Mahmood & Ramezani, Hossein, 2024. "Enhancing cybersecurity in wind turbines: A resilient reinforcement learning-based optimal control for mitigating FDI attacks," Applied Energy, Elsevier, vol. 373(C).
    18. Zhao, Jing & Yang, Zilan & Shi, Linyu & Liu, Dehan & Li, Haonan & Mi, Yumiao & Wang, Hongbin & Feng, Meili & Hutagaol, Timothy Joseph, 2024. "Photovoltaic capacity dynamic tracking model predictive control strategy of air-conditioning systems with consideration of flexible loads," Applied Energy, Elsevier, vol. 356(C).
    19. Wu, Chunying & Sun, Lingfang & Piao, Heng & Yao, Lijia, 2024. "Adaptive fuzzy finite time integral sliding mode control of the coordinated system for 350 MW supercritical once-through boiler unit to enhance flexibility," Energy, Elsevier, vol. 302(C).
    20. Shi, Jie & Wang, Luhao & Lee, Wei-Jen & Cheng, Xingong & Zong, Xiju, 2019. "Hybrid Energy Storage System (HESS) optimization enabling very short-term wind power generation scheduling based on output feature extraction," Applied Energy, Elsevier, vol. 256(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:391:y:2025:i:c:s030626192500580x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.