IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v264y2023ics036054422203095x.html
   My bibliography  Save this article

Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning

Author

Listed:
  • Qin, Haosen
  • Yu, Zhen
  • Li, Tailu
  • Liu, Xueliang
  • Li, Li

Abstract

Controlling Heating, Ventilation and Air Conditioning (HVAC) systems is critical to improving energy efficiency of demand-side. In this paper, a model-free optimal control method based on deep reinforcement learning is proposed to control the heat pump start/stop and room temperature setting in residential buildings. The optimization goal of this method is to obtain the highest comprehensive reward which considering thermal comfort and energy cost. Firstly, the randomness, learning process, thermal comfort and energy consumption of the model-free controller are systematically investigated by a simulation system based on measured data. The results show that randomness has a significant impact on the initial performance and convergence speed of the model-free controller; The model-free controller has a linear accumulation of comprehensive rewards during the learning process, and the slope of the accumulated comprehensive rewards can be used to determine whether the controller converges; The model-free controller coordinates monitoring data, weather forecasts and building thermal inertia to achieve the highest comprehensive reward. Afterwards, the model-free controller was verified in a nearly zero energy residential building in Beijing, China. The results show that model-free controller improves the comprehensive reward by 15.3% compared to rule-based method.

Suggested Citation

  • Qin, Haosen & Yu, Zhen & Li, Tailu & Liu, Xueliang & Li, Li, 2023. "Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning," Energy, Elsevier, vol. 264(C).
  • Handle: RePEc:eee:energy:v:264:y:2023:i:c:s036054422203095x
    DOI: 10.1016/j.energy.2022.126209
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S036054422203095X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.energy.2022.126209?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kim, Wonuk & Jeon, Yongseok & Kim, Yongchan, 2016. "Simulation-based optimization of an integrated daylighting and HVAC system using the design of experiments method," Applied Energy, Elsevier, vol. 162(C), pages 666-674.
    2. Chen, Yongbao & Xu, Peng & Chu, Yiyi & Li, Weilin & Wu, Yuntao & Ni, Lizhou & Bao, Yi & Wang, Kun, 2017. "Short-term electrical load forecasting using the Support Vector Regression (SVR) model to calculate the demand response baseline for office buildings," Applied Energy, Elsevier, vol. 195(C), pages 659-670.
    3. Zou, Runnan & Fan, Likang & Dong, Yanrui & Zheng, Siyu & Hu, Chenxing, 2021. "DQL energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle," Energy, Elsevier, vol. 225(C).
    4. Liu, Teng & Wang, Bo & Yang, Chenglang, 2018. "Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning," Energy, Elsevier, vol. 160(C), pages 544-555.
    5. Feng, Yanxiao & Duan, Qiuhua & Chen, Xi & Yakkali, Sai Santosh & Wang, Julian, 2021. "Space cooling energy usage prediction based on utility data for residential buildings using machine learning methods," Applied Energy, Elsevier, vol. 291(C).
    6. Haosen Qin & Zhen Yu & Tailu Li & Xueliang Liu & Li Li, 2022. "Heating Control Strategy Based on Dynamic Programming for Building Energy Saving and Emission Reduction," IJERPH, MDPI, vol. 19(21), pages 1-27, October.
    7. Bischi, Aldo & Taccari, Leonardo & Martelli, Emanuele & Amaldi, Edoardo & Manzolini, Giampaolo & Silva, Paolo & Campanari, Stefano & Macchi, Ennio, 2014. "A detailed MILP optimization model for combined cooling, heat and power system operation planning," Energy, Elsevier, vol. 74(C), pages 12-26.
    8. Kusiak, Andrew & Li, Mingyang & Tang, Fan, 2010. "Modeling and optimization of HVAC energy consumption," Applied Energy, Elsevier, vol. 87(10), pages 3092-3102, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zhou, Kaile & Peng, Ning & Yin, Hui & Hu, Rong, 2023. "Urban virtual power plant operation optimization with incentive-based demand response," Energy, Elsevier, vol. 282(C).
    2. Vladimir Franki & Darin Majnarić & Alfredo Višković, 2023. "A Comprehensive Review of Artificial Intelligence (AI) Companies in the Power Sector," Energies, MDPI, vol. 16(3), pages 1-35, January.
    3. Elsisi, Mahmoud & Amer, Mohammed & Dababat, Alya’ & Su, Chun-Lien, 2023. "A comprehensive review of machine learning and IoT solutions for demand side energy management, conservation, and resilient operation," Energy, Elsevier, vol. 281(C).
    4. Jan Wrana & Wojciech Struzik & Katarzyna Jaromin-Gleń & Piotr Gleń, 2023. "FCH HVAC Honeycomb Ring Network—Transition from Traditional Power Supply Systems in Existing and Revitalized Areas," Energies, MDPI, vol. 16(24), pages 1-14, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cui, Can & Zhang, Xin & Cai, Wenjian, 2020. "An energy-saving oriented air balancing method for demand controlled ventilation systems with branch and black-box model," Applied Energy, Elsevier, vol. 264(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Afroz, Zakia & Shafiullah, GM & Urmee, Tania & Higgins, Gary, 2018. "Modeling techniques used in building HVAC control systems: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 83(C), pages 64-84.
    4. Jing, Gang & Cai, Wenjian & Zhang, Xin & Cui, Can & Liu, Hongwu & Wang, Cheng, 2020. "An energy-saving control strategy for multi-zone demand controlled ventilation system with data-driven model and air balancing control," Energy, Elsevier, vol. 199(C).
    5. Yang, Shiyu & Wan, Man Pun & Chen, Wanyu & Ng, Bing Feng & Dubey, Swapnil, 2020. "Model predictive control with adaptive machine-learning-based model for building energy efficiency and comfort optimization," Applied Energy, Elsevier, vol. 271(C).
    6. Haosen Qin & Zhen Yu & Tailu Li & Xueliang Liu & Li Li, 2022. "Heating Control Strategy Based on Dynamic Programming for Building Energy Saving and Emission Reduction," IJERPH, MDPI, vol. 19(21), pages 1-27, October.
    7. Chen, Yongbao & Chen, Zhe & Xu, Peng & Li, Weilin & Sha, Huajing & Yang, Zhiwei & Li, Guowen & Hu, Chonghe, 2019. "Quantification of electricity flexibility in demand response: Office building case study," Energy, Elsevier, vol. 188(C).
    8. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    9. Guelpa, Elisa & Bischi, Aldo & Verda, Vittorio & Chertkov, Michael & Lund, Henrik, 2019. "Towards future infrastructures for sustainable multi-energy systems: A review," Energy, Elsevier, vol. 184(C), pages 2-21.
    10. Rongjiang Ma & Xianlin Wang & Ming Shan & Nanyang Yu & Shen Yang, 2020. "Recognition of Variable-Speed Equipment in an Air-Conditioning System Using Numerical Analysis of Energy-Consumption Data," Energies, MDPI, vol. 13(18), pages 1-14, September.
    11. Matteo Acquarone & Claudio Maino & Daniela Misul & Ezio Spessa & Antonio Mastropietro & Luca Sorrentino & Enrico Busto, 2023. "Influence of the Reward Function on the Selection of Reinforcement Learning Agents for Hybrid Electric Vehicles Real-Time Control," Energies, MDPI, vol. 16(6), pages 1-22, March.
    12. Bingjie Jin & Guihua Zeng & Zhilin Lu & Hongqiao Peng & Shuxin Luo & Xinhe Yang & Haojun Zhu & Mingbo Liu, 2022. "Hybrid LSTM–BPNN-to-BPNN Model Considering Multi-Source Information for Forecasting Medium- and Long-Term Electricity Peak Load," Energies, MDPI, vol. 15(20), pages 1-20, October.
    13. Kazmi, Hussain & Suykens, Johan & Balint, Attila & Driesen, Johan, 2019. "Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads," Applied Energy, Elsevier, vol. 238(C), pages 1022-1035.
    14. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    15. Tsoumalis, Georgios I. & Bampos, Zafeirios N. & Biskas, Pandelis N. & Keranidis, Stratos D. & Symeonidis, Polychronis A. & Voulgarakis, Dimitrios K., 2022. "A novel system for providing explicit demand response from domestic natural gas boilers," Applied Energy, Elsevier, vol. 317(C).
    16. Pérez-Iribarren, E. & González-Pino, I. & Azkorra-Larrinaga, Z. & Gómez-Arriarán, I., 2020. "Optimal design and operation of thermal energy storage systems in micro-cogeneration plants," Applied Energy, Elsevier, vol. 265(C).
    17. Louback, Eduardo & Biswas, Atriya & Machado, Fabricio & Emadi, Ali, 2024. "A review of the design process of energy management systems for dual-motor battery electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 193(C).
    18. Kang, Ligai & Yang, Junhong & An, Qingsong & Deng, Shuai & Zhao, Jun & Wang, Hui & Li, Zelin, 2017. "Effects of load following operational strategy on CCHP system with an auxiliary ground source heat pump considering carbon tax and electricity feed in tariff," Applied Energy, Elsevier, vol. 194(C), pages 454-466.
    19. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    20. Xiaofeng Liu & Shijun Wang & Jiawen Sun, 2018. "Energy Management for Community Energy Network with CHP Based on Cooperative Game," Energies, MDPI, vol. 11(5), pages 1-18, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:264:y:2023:i:c:s036054422203095x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.