IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v390y2025ics030626192500546x.html
   My bibliography  Save this article

Year-round operational optimization of HVAC systems using hierarchical deep reinforcement learning for enhancing indoor air quality and reducing energy consumption

Author

Listed:
  • Liao, Chenxin
  • Miyata, Shohei
  • Qu, Ming
  • Akashi, Yasunori

Abstract

Balancing user comfort demands with energy consumption in heating, ventilation, and air conditioning (HVAC) systems is a critical challenge in the field of control optimization. Currently, deep reinforcement learning (DRL) algorithms have achieved some success in optimizing HVAC system control. However, training agents with existing algorithms becomes challenging, when dealing with the hybrid action control problem. This study proposes an HVAC system optimization control method based on hierarchical deep reinforcement learning (HDRL), which aims to minimize system energy consumption while ensuring indoor comfort and maintaining air quality. The proposed HDRL method employs a two-layer agent architecture, where the bottom agent is responsible for optimizing the frequency control of fans and pumps and the top agent flexibly selects the daily operating mode based on outdoor meteorological conditions. We designed a multi-objective reward function based on Gaussian distribution, hyperbolic tangent function, energy consumption normalization for bottom agents, and temperature-based reward function for the top agent to achieve the reliable training of the agents. Experimental results indicated that the HDRL method significantly improves indoor temperature and CO2 concentration control during the year-round operation of the HVAC system. Compared to conventional PI control, the temperature compliance rate is increased by 8.37 %, and the combined temperature and CO2 compliance rate is improved by 7.8 %, while achieving a 3.95 % reduction in energy consumption.

Suggested Citation

  • Liao, Chenxin & Miyata, Shohei & Qu, Ming & Akashi, Yasunori, 2025. "Year-round operational optimization of HVAC systems using hierarchical deep reinforcement learning for enhancing indoor air quality and reducing energy consumption," Applied Energy, Elsevier, vol. 390(C).
    • Handle: RePEc:eee:appene:v:390:y:2025:i:c:s030626192500546x
    DOI: 10.1016/j.apenergy.2025.125816
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S030626192500546X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125816?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    2. Wang, Hao & Chen, Xiwen & Vital, Natan & Duffy, Edward & Razi, Abolfazl, 2024. "Energy optimization for HVAC systems in multi-VAV open offices: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 356(C).
    3. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    4. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Operational optimization for off-grid renewable building energy system using deep reinforcement learning," Applied Energy, Elsevier, vol. 325(C).
    5. Gao, Yuan & Miyata, Shohei & Akashi, Yasunori, 2023. "Energy saving and indoor temperature control for an office building using tube-based robust model predictive control," Applied Energy, Elsevier, vol. 341(C).
    6. Gao, Yuan & Shi, Shanrui & Miyata, Shohei & Akashi, Yasunori, 2024. "Successful application of predictive information in deep reinforcement learning control: A case study based on an office building HVAC system," Energy, Elsevier, vol. 291(C).
    7. Lei, Yue & Zhan, Sicheng & Ono, Eikichi & Peng, Yuzhen & Zhang, Zhiang & Hasama, Takamasa & Chong, Adrian, 2022. "A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings," Applied Energy, Elsevier, vol. 324(C).
    8. Nam, KiJeon & Heo, SungKu & Li, Qian & Loy-Benitez, Jorge & Kim, MinJeong & Park, DuckShin & Yoo, ChangKyoo, 2020. "A proactive energy-efficient optimal ventilation system using artificial intelligent techniques under outdoor air quality conditions," Applied Energy, Elsevier, vol. 266(C).
    9. Kong, Meng & Dong, Bing & Zhang, Rongpeng & O'Neill, Zheng, 2022. "HVAC energy savings, thermal comfort and air quality for occupant-centric control through a side-by-side experimental study," Applied Energy, Elsevier, vol. 306(PA).
    10. Shi, Shanrui & Miyata, Shohei & Akashi, Yasunori, 2025. "Event-driven model-based optimal demand-controlled ventilation for multizone VAV systems: Enhancing energy efficiency and indoor environmental quality," Applied Energy, Elsevier, vol. 377(PD).
    11. Hu, Zehuan & Gao, Yuan & Sun, Luning & Mae, Masayuki & Imaizumi, Taiji, 2024. "Improved robust model predictive control for residential building air conditioning and photovoltaic power generation with battery energy storage system under weather forecast uncertainty," Applied Energy, Elsevier, vol. 371(C).
    12. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    13. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    14. Singh, A. & Syal, M. & Grady, S.C. & Korkmaz, S., 2010. "Effects of green buildings on employee health and productivity," American Journal of Public Health, American Public Health Association, vol. 100(9), pages 1665-1668.
    15. Gao, Yuan & Hu, Zehuan & Shi, Shanrui & Chen, Wei-An & Liu, Mingzhe, 2024. "Adversarial discriminative domain adaptation for solar radiation prediction: A cross-regional study for zero-label transfer learning in Japan," Applied Energy, Elsevier, vol. 359(C).
    16. Jayathissa, P. & Luzzatto, M. & Schmidli, J. & Hofer, J. & Nagy, Z. & Schlueter, A., 2017. "Optimising building net energy demand with dynamic BIPV shading," Applied Energy, Elsevier, vol. 202(C), pages 726-735.
    17. Liu, Mingzhe & Guo, Mingyue & Fu, Yangyang & O’Neill, Zheng & Gao, Yuan, 2024. "Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications," Applied Energy, Elsevier, vol. 372(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    2. Gao, Yuan & Hu, Zehuan & Yamate, Shun & Otomo, Junichiro & Chen, Wei-An & Liu, Mingzhe & Xu, Tingting & Ruan, Yingjun & Shang, Juan, 2025. "Unlocking predictive insights and interpretability in deep reinforcement learning for Building-Integrated Photovoltaic and Battery (BIPVB) systems," Applied Energy, Elsevier, vol. 384(C).
    3. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    4. Wang, Xuezheng & Dong, Bing, 2024. "Long-term experimental evaluation and comparison of advanced controls for HVAC systems," Applied Energy, Elsevier, vol. 371(C).
    5. Gao, Yuan & Liu, Mingzhe & Hu, Zehuan & Yamate, Shun & Otomo, Junichiro & Chen, Wei-An & O’Neill, Zheng, 2025. "Quantitative analysis of energy justice in demand response: Insights from real residential data in Texas, USA," Renewable Energy, Elsevier, vol. 242(C).
    6. Liu, Yuntao & Song, Yutong & Cui, Can, 2025. "Towards smart control and energy efficiency for multi-zone ventilation systems via an imitation-interaction learning method in energy-aware buildings," Energy, Elsevier, vol. 314(C).
    7. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    8. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    9. Heidari, Amirreza & Girardin, Luc & Dorsaz, Cédric & Maréchal, François, 2025. "A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation," Applied Energy, Elsevier, vol. 378(PA).
    10. Panagiotis Michailidis & Iakovos Michailidis & Elias Kosmatopoulos, 2025. "Reinforcement Learning for Optimizing Renewable Energy Utilization in Buildings: A Review on Applications and Innovations," Energies, MDPI, vol. 18(7), pages 1-40, March.
    11. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    12. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    13. Liu, Mingzhe & Guo, Mingyue & Fu, Yangyang & O’Neill, Zheng & Gao, Yuan, 2024. "Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications," Applied Energy, Elsevier, vol. 372(C).
    14. Zhang, Bin & Hu, Weihao & Ghias, Amer M.Y.M. & Xu, Xiao & Chen, Zhe, 2022. "Multi-agent deep reinforcement learning-based coordination control for grid-aware multi-buildings," Applied Energy, Elsevier, vol. 328(C).
    15. Deng, Xiangtian & Zhang, Yi & Jiang, Yi & Zhang, Yi & Qi, He, 2024. "A novel operation method for renewable building by combining distributed DC energy system and deep reinforcement learning," Applied Energy, Elsevier, vol. 353(PB).
    16. Guo, Yuxiang & Qu, Shengli & Wang, Chuang & Xing, Ziwen & Duan, Kaiwen, 2024. "Optimal dynamic thermal management for data center via soft actor-critic algorithm with dynamic control interval and combined-value state space," Applied Energy, Elsevier, vol. 373(C).
    17. Dalia Mohammed Talat Ebrahim Ali & Violeta Motuzienė & Rasa Džiugaitė-Tumėnienė, 2024. "AI-Driven Innovations in Building Energy Management Systems: A Review of Potential Applications and Energy Savings," Energies, MDPI, vol. 17(17), pages 1-35, August.
    18. Chen, Qi & Kuang, Zhonghong & Liu, Xiaohua & Zhang, Tao, 2024. "Application-oriented assessment of grid-connected PV-battery system with deep reinforcement learning in buildings considering electricity price dynamics," Applied Energy, Elsevier, vol. 364(C).
    19. Haikui Jin & Jian Wang & Ying Wang & Yingjun Ruan & Yuan Gao & Fanyue Qian & Xiaoyan Xu & Chen Ju & Xun Dong, 2025. "Adaptability Study of Hydrogen Fuel Cell Integrated Energy Systems," Energies, MDPI, vol. 18(8), pages 1-20, April.
    20. Xu, Wenjie & Svetozarevic, Bratislav & Di Natale, Loris & Heer, Philipp & Jones, Colin N., 2024. "Data-driven adaptive building thermal controller tuning with constraints: A primal–dual contextual Bayesian optimization approach," Applied Energy, Elsevier, vol. 358(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:390:y:2025:i:c:s030626192500546x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.