IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v323y2025ics0360544225014665.html
   My bibliography  Save this article

Optimal control of HVAC systems through active disturbance rejection control-assisted reinforcement learning

Author

Listed:
  • Cui, Can
  • Xue, Jiahui
  • Liu, Lanjun

Abstract

Optimal control of multi-zone HVAC systems may suffer from noise and disturbances that affect control accuracy and performance, and faces computational challenges caused by multiple control variables. To address these challenges, this paper proposes a novel method that incorporates reinforcement learning and active disturbance rejection control through a main-auxiliary controller configuration. A main controller is designed based on twin delayed deep deterministic policy gradient, which is responsible for controlling zone supply airflows. An auxiliary controller is configured based on active disturbance rejection control, which regulates the fresh air ratio and meanwhile handling the disturbances and uncertainties. The two controllers work in parallel with exchange information in real-time to optimize HVAC systems in dynamically uncertain environments. In the proposed method, the control variables are separated and handled by main and auxiliary controllers respectively, which reduces the action space of reinforcement learning algorithm and partly decouples the thermal loads and ventilation loads. An EnergyPlus-Python co-simulation platform has been developed using real-world data. Test results demonstrate that the proposed AD-RL method can enhance indoor comfort and IAQ. Furthermore, compared to the rule-based method and the classical TD3-based approach, it can reduce the daily HVAC energy consumption by up to 22.37 % and 13.53 %, respectively.

Suggested Citation

  • Cui, Can & Xue, Jiahui & Liu, Lanjun, 2025. "Optimal control of HVAC systems through active disturbance rejection control-assisted reinforcement learning," Energy, Elsevier, vol. 323(C).
    • Handle: RePEc:eee:energy:v:323:y:2025:i:c:s0360544225014665
    DOI: 10.1016/j.energy.2025.135824
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544225014665
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2025.135824?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Široký, Jan & Oldewurtel, Frauke & Cigler, Jiří & Prívara, Samuel, 2011. "Experimental analysis of model predictive control for an energy efficient building heating system," Applied Energy, Elsevier, vol. 88(9), pages 3079-3087.
    2. Sun, Li & Jin, Yuhui & You, Fengqi, 2020. "Active disturbance rejection temperature control of open-cathode proton exchange membrane fuel cell," Applied Energy, Elsevier, vol. 261(C).
    3. Wang, Hao & Chen, Xiwen & Vital, Natan & Duffy, Edward & Razi, Abolfazl, 2024. "Energy optimization for HVAC systems in multi-VAV open offices: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 356(C).
    4. Jiang, Yuliang & Zhu, Shanying & Xu, Qimin & Yang, Bo & Guan, Xinping, 2023. "Hybrid modeling-based temperature and humidity adaptive control for a multi-zone HVAC system," Applied Energy, Elsevier, vol. 334(C).
    5. Kim, Wonuk & Jeon, Seung Won & Kim, Yongchan, 2016. "Model-based multi-objective optimal control of a VRF (variable refrigerant flow) combined system with DOAS (dedicated outdoor air system) using genetic algorithm under heating conditions," Energy, Elsevier, vol. 107(C), pages 196-204.
    6. Song, Kwonsik & Jang, Youjin & Park, Moonseo & Lee, Hyun-Soo & Ahn, Joseph, 2020. "Energy efficiency of end-user groups for personalized HVAC control in multi-zone buildings," Energy, Elsevier, vol. 206(C).
    7. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    8. Goyal, Siddharth & Barooah, Prabir & Middelkoop, Timothy, 2015. "Experimental study of occupancy-based control of HVAC zones," Applied Energy, Elsevier, vol. 140(C), pages 75-84.
    9. Joe, Jaewan & Im, Piljae & Cui, Borui & Dong, Jin, 2023. "Model-based predictive control of multi-zone commercial building with a lumped building modelling approach," Energy, Elsevier, vol. 263(PA).
    10. Li, Wenzhuo & Wang, Shengwei, 2020. "A multi-agent based distributed approach for optimal control of multi-zone ventilation systems considering indoor air quality and energy use," Applied Energy, Elsevier, vol. 275(C).
    11. Gao, Yixiang & Li, Shuhui & Fu, Xingang & Dong, Weizhen & Lu, Bing & Li, Zhongwen, 2020. "Energy management and demand response with intelligent learning for multi-thermal-zone buildings," Energy, Elsevier, vol. 210(C).
    12. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    13. Yang, Ting & Zhao, Liyuan & Li, Wei & Wu, Jianzhong & Zomaya, Albert Y., 2021. "Towards healthy and cost-effective indoor environment management in smart homes: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 300(C).
    14. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    15. Giampieri, A. & Ma, Z. & Ling-Chin, J. & Roskilly, A.P. & Smallbone, A.J., 2022. "An overview of solutions for airborne viral transmission reduction related to HVAC systems including liquid desiccant air-scrubbing," Energy, Elsevier, vol. 244(PA).
    16. Wu, Zhenlong & Liu, Yanhong & Li, Donghai & Chen, YangQuan, 2023. "Multivariable active disturbance rejection control for compression liquid chiller system," Energy, Elsevier, vol. 262(PA).
    17. Jin, Xiaolong & Mu, Yunfei & Jia, Hongjie & Wu, Jianzhong & Jiang, Tao & Yu, Xiaodan, 2017. "Dynamic economic dispatch of a hybrid energy microgrid considering building based virtual energy storage system," Applied Energy, Elsevier, vol. 194(C), pages 386-398.
    18. Fiorentini, Massimo & Wall, Josh & Ma, Zhenjun & Braslavsky, Julio H. & Cooper, Paul, 2017. "Hybrid model predictive control of a residential HVAC system with on-site thermal energy generation and storage," Applied Energy, Elsevier, vol. 187(C), pages 465-479.
    19. Turanjanin, Valentina & Vučićević, Biljana & Jovanović, Marina & Mirkov, Nikola & Lazović, Ivan, 2014. "Indoor CO2 measurements in Serbian schools and ventilation rate calculation," Energy, Elsevier, vol. 77(C), pages 290-296.
    20. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    21. Fang, Xi & Gong, Guangcai & Li, Guannan & Chun, Liang & Peng, Pei & Li, Wenqiang & Shi, Xing, 2023. "Cross temporal-spatial transferability investigation of deep reinforcement learning control strategy in the building HVAC system level," Energy, Elsevier, vol. 263(PB).
    22. Pang, Zhihong & Chen, Yan & Zhang, Jian & O'Neill, Zheng & Cheng, Hwakong & Dong, Bing, 2020. "Nationwide HVAC energy-saving potential quantification for office buildings with occupant-centric controls in various climates," Applied Energy, Elsevier, vol. 279(C).
    23. Wang, Qianlin & Han, Jiaqi & Chen, Feng & Hu, Su & Yun, Cheng & Dou, Zhan & Yan, Tingjun & Yang, Guoan, 2024. "Modeling risk characterization networks for chemical processes based on multi-variate data," Energy, Elsevier, vol. 293(C).
    24. Du, Yan & Zandi, Helia & Kotevska, Olivera & Kurte, Kuldeep & Munk, Jeffery & Amasyali, Kadir & Mckee, Evan & Li, Fangxing, 2021. "Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning," Applied Energy, Elsevier, vol. 281(C).
    25. Yu, Xiaobing & Lu, Yangchen, 2023. "Reinforcement learning-based multi-objective differential evolution for wind farm layout optimization," Energy, Elsevier, vol. 284(C).
    26. He, Hongwen & Su, Qicong & Huang, Ruchen & Niu, Zegong, 2024. "Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm," Energy, Elsevier, vol. 294(C).
    27. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    2. Liu, Yuntao & Cui, Can, 2024. "A bi-level real-time optimal control strategy for thermal coupled multi-zone dedicated outside air system-assisted HVAC systems," Energy, Elsevier, vol. 306(C).
    3. Gianluca Serale & Massimo Fiorentini & Alfonso Capozzoli & Daniele Bernardini & Alberto Bemporad, 2018. "Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities," Energies, MDPI, vol. 11(3), pages 1-35, March.
    4. Fang, Xi & Gong, Guangcai & Li, Guannan & Chun, Liang & Peng, Pei & Li, Wenqiang & Shi, Xing, 2023. "Cross temporal-spatial transferability investigation of deep reinforcement learning control strategy in the building HVAC system level," Energy, Elsevier, vol. 263(PB).
    5. Raman, Naren Srivaths & Chen, Bo & Barooah, Prabir, 2022. "On energy-efficient HVAC operation with Model Predictive Control: A multiple climate zone study," Applied Energy, Elsevier, vol. 324(C).
    6. Yang, Shiyu & Wan, Man Pun & Ng, Bing Feng & Dubey, Swapnil & Henze, Gregor P. & Chen, Wanyu & Baskaran, Krishnamoorthy, 2020. "Experimental study of model predictive control for an air-conditioning system with dedicated outdoor air system," Applied Energy, Elsevier, vol. 257(C).
    7. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    8. Guo, Fangzhou & Ham, Sang woo & Kim, Donghun & Moon, Hyeun Jun, 2025. "Deep reinforcement learning control for co-optimizing energy consumption, thermal comfort, and indoor air quality in an office building," Applied Energy, Elsevier, vol. 377(PA).
    9. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    10. Liu, Shuo & Liu, Xiaohua & Zhang, Tao & Wang, Chaoliang & Liu, Wei, 2024. "Joint optimization for temperature and humidity independent control system based on multi-agent reinforcement learning with cooperative mechanisms," Applied Energy, Elsevier, vol. 375(C).
    11. Li, Jianwei & Liu, Jie & Yang, Qingqing & Wang, Tianci & He, Hongwen & Wang, Hanxiao & Sun, Fengchun, 2025. "Reinforcement learning based energy management for fuel cell hybrid electric vehicles: A comprehensive review on decision process reformulation and strategy implementation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 213(C).
    12. Lu, Ruyuan & Li, Xin & Chen, Ronghao & Lei, Aimin & Ma, Xiaoming, 2024. "An Alternative Reinforcement Learning (ARL) control strategy for data center air-cooled HVAC systems," Energy, Elsevier, vol. 308(C).
    13. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    14. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).
    15. Tang, Tianfeng & Peng, Qianlong & Shi, Qing & Peng, Qingguo & Zhao, Jin & Chen, Chaoyi & Wang, Guangwei, 2024. "Energy management of fuel cell hybrid electric bus in mountainous regions: A deep reinforcement learning approach considering terrain characteristics," Energy, Elsevier, vol. 311(C).
    16. Liao, Chenxin & Miyata, Shohei & Qu, Ming & Akashi, Yasunori, 2025. "Year-round operational optimization of HVAC systems using hierarchical deep reinforcement learning for enhancing indoor air quality and reducing energy consumption," Applied Energy, Elsevier, vol. 390(C).
    17. Fan, Songli & Xu, Guodong & Jiang, Baoping & Wu, Zhengtian & Xing, Haijun & Gao, Yang & Ai, Qian, 2025. "Low-carbon economic operation of commercial park-level integrated energy systems incorporating supply and demand flexibility," Energy, Elsevier, vol. 323(C).
    18. Zhang, Limao & Chen, Chao & Zhou, Cheng & Luo, Yongqiang & Wu, Xiaoying, 2025. "Zone-based many-objective building decarbonization considering outdoor temperature and occupation uncertainty," Renewable and Sustainable Energy Reviews, Elsevier, vol. 208(C).
    19. Dalia Mohammed Talat Ebrahim Ali & Violeta Motuzienė & Rasa Džiugaitė-Tumėnienė, 2024. "AI-Driven Innovations in Building Energy Management Systems: A Review of Potential Applications and Energy Savings," Energies, MDPI, vol. 17(17), pages 1-35, August.
    20. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:323:y:2025:i:c:s0360544225014665. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.