IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v264y2023ics036054422203095x.html
   My bibliography  Save this article

Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning

Author

Listed:
  • Qin, Haosen
  • Yu, Zhen
  • Li, Tailu
  • Liu, Xueliang
  • Li, Li

Abstract

Controlling Heating, Ventilation and Air Conditioning (HVAC) systems is critical to improving energy efficiency of demand-side. In this paper, a model-free optimal control method based on deep reinforcement learning is proposed to control the heat pump start/stop and room temperature setting in residential buildings. The optimization goal of this method is to obtain the highest comprehensive reward which considering thermal comfort and energy cost. Firstly, the randomness, learning process, thermal comfort and energy consumption of the model-free controller are systematically investigated by a simulation system based on measured data. The results show that randomness has a significant impact on the initial performance and convergence speed of the model-free controller; The model-free controller has a linear accumulation of comprehensive rewards during the learning process, and the slope of the accumulated comprehensive rewards can be used to determine whether the controller converges; The model-free controller coordinates monitoring data, weather forecasts and building thermal inertia to achieve the highest comprehensive reward. Afterwards, the model-free controller was verified in a nearly zero energy residential building in Beijing, China. The results show that model-free controller improves the comprehensive reward by 15.3% compared to rule-based method.

Suggested Citation

  • Qin, Haosen & Yu, Zhen & Li, Tailu & Liu, Xueliang & Li, Li, 2023. "Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning," Energy, Elsevier, vol. 264(C).
  • Handle: RePEc:eee:energy:v:264:y:2023:i:c:s036054422203095x
    DOI: 10.1016/j.energy.2022.126209
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S036054422203095X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.energy.2022.126209?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kim, Wonuk & Jeon, Yongseok & Kim, Yongchan, 2016. "Simulation-based optimization of an integrated daylighting and HVAC system using the design of experiments method," Applied Energy, Elsevier, vol. 162(C), pages 666-674.
    2. Chen, Yongbao & Xu, Peng & Chu, Yiyi & Li, Weilin & Wu, Yuntao & Ni, Lizhou & Bao, Yi & Wang, Kun, 2017. "Short-term electrical load forecasting using the Support Vector Regression (SVR) model to calculate the demand response baseline for office buildings," Applied Energy, Elsevier, vol. 195(C), pages 659-670.
    3. Zou, Runnan & Fan, Likang & Dong, Yanrui & Zheng, Siyu & Hu, Chenxing, 2021. "DQL energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle," Energy, Elsevier, vol. 225(C).
    4. Liu, Teng & Wang, Bo & Yang, Chenglang, 2018. "Online Markov Chain-based energy management for a hybrid tracked vehicle with speedy Q-learning," Energy, Elsevier, vol. 160(C), pages 544-555.
    5. Feng, Yanxiao & Duan, Qiuhua & Chen, Xi & Yakkali, Sai Santosh & Wang, Julian, 2021. "Space cooling energy usage prediction based on utility data for residential buildings using machine learning methods," Applied Energy, Elsevier, vol. 291(C).
    6. Haosen Qin & Zhen Yu & Tailu Li & Xueliang Liu & Li Li, 2022. "Heating Control Strategy Based on Dynamic Programming for Building Energy Saving and Emission Reduction," IJERPH, MDPI, vol. 19(21), pages 1-27, October.
    7. Bischi, Aldo & Taccari, Leonardo & Martelli, Emanuele & Amaldi, Edoardo & Manzolini, Giampaolo & Silva, Paolo & Campanari, Stefano & Macchi, Ennio, 2014. "A detailed MILP optimization model for combined cooling, heat and power system operation planning," Energy, Elsevier, vol. 74(C), pages 12-26.
    8. Kusiak, Andrew & Li, Mingyang & Tang, Fan, 2010. "Modeling and optimization of HVAC energy consumption," Applied Energy, Elsevier, vol. 87(10), pages 3092-3102, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vladimir Franki & Darin Majnarić & Alfredo Višković, 2023. "A Comprehensive Review of Artificial Intelligence (AI) Companies in the Power Sector," Energies, MDPI, vol. 16(3), pages 1-35, January.
    2. Elsisi, Mahmoud & Amer, Mohammed & Dababat, Alya’ & Su, Chun-Lien, 2023. "A comprehensive review of machine learning and IoT solutions for demand side energy management, conservation, and resilient operation," Energy, Elsevier, vol. 281(C).
    3. Jan Wrana & Wojciech Struzik & Katarzyna Jaromin-Gleń & Piotr Gleń, 2023. "FCH HVAC Honeycomb Ring Network—Transition from Traditional Power Supply Systems in Existing and Revitalized Areas," Energies, MDPI, vol. 16(24), pages 1-14, December.
    4. Zhou, Kaile & Peng, Ning & Yin, Hui & Hu, Rong, 2023. "Urban virtual power plant operation optimization with incentive-based demand response," Energy, Elsevier, vol. 282(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cui, Can & Zhang, Xin & Cai, Wenjian, 2020. "An energy-saving oriented air balancing method for demand controlled ventilation systems with branch and black-box model," Applied Energy, Elsevier, vol. 264(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Yang, Shiyu & Wan, Man Pun & Chen, Wanyu & Ng, Bing Feng & Dubey, Swapnil, 2020. "Model predictive control with adaptive machine-learning-based model for building energy efficiency and comfort optimization," Applied Energy, Elsevier, vol. 271(C).
    4. Afroz, Zakia & Shafiullah, GM & Urmee, Tania & Higgins, Gary, 2018. "Modeling techniques used in building HVAC control systems: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 83(C), pages 64-84.
    5. Jing, Gang & Cai, Wenjian & Zhang, Xin & Cui, Can & Liu, Hongwu & Wang, Cheng, 2020. "An energy-saving control strategy for multi-zone demand controlled ventilation system with data-driven model and air balancing control," Energy, Elsevier, vol. 199(C).
    6. Haosen Qin & Zhen Yu & Tailu Li & Xueliang Liu & Li Li, 2022. "Heating Control Strategy Based on Dynamic Programming for Building Energy Saving and Emission Reduction," IJERPH, MDPI, vol. 19(21), pages 1-27, October.
    7. Chen, Yongbao & Chen, Zhe & Xu, Peng & Li, Weilin & Sha, Huajing & Yang, Zhiwei & Li, Guowen & Hu, Chonghe, 2019. "Quantification of electricity flexibility in demand response: Office building case study," Energy, Elsevier, vol. 188(C).
    8. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    9. Guelpa, Elisa & Bischi, Aldo & Verda, Vittorio & Chertkov, Michael & Lund, Henrik, 2019. "Towards future infrastructures for sustainable multi-energy systems: A review," Energy, Elsevier, vol. 184(C), pages 2-21.
    10. Rongjiang Ma & Xianlin Wang & Ming Shan & Nanyang Yu & Shen Yang, 2020. "Recognition of Variable-Speed Equipment in an Air-Conditioning System Using Numerical Analysis of Energy-Consumption Data," Energies, MDPI, vol. 13(18), pages 1-14, September.
    11. Matteo Acquarone & Claudio Maino & Daniela Misul & Ezio Spessa & Antonio Mastropietro & Luca Sorrentino & Enrico Busto, 2023. "Influence of the Reward Function on the Selection of Reinforcement Learning Agents for Hybrid Electric Vehicles Real-Time Control," Energies, MDPI, vol. 16(6), pages 1-22, March.
    12. Bingjie Jin & Guihua Zeng & Zhilin Lu & Hongqiao Peng & Shuxin Luo & Xinhe Yang & Haojun Zhu & Mingbo Liu, 2022. "Hybrid LSTM–BPNN-to-BPNN Model Considering Multi-Source Information for Forecasting Medium- and Long-Term Electricity Peak Load," Energies, MDPI, vol. 15(20), pages 1-20, October.
    13. Kang, Ligai & Yang, Junhong & An, Qingsong & Deng, Shuai & Zhao, Jun & Wang, Hui & Li, Zelin, 2017. "Effects of load following operational strategy on CCHP system with an auxiliary ground source heat pump considering carbon tax and electricity feed in tariff," Applied Energy, Elsevier, vol. 194(C), pages 454-466.
    14. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    15. Xiaofeng Liu & Shijun Wang & Jiawen Sun, 2018. "Energy Management for Community Energy Network with CHP Based on Cooperative Game," Energies, MDPI, vol. 11(5), pages 1-18, April.
    16. Wang, Kejun & Qi, Xiaoxia & Liu, Hongda & Song, Jiakang, 2018. "Deep belief network based k-means cluster approach for short-term wind power forecasting," Energy, Elsevier, vol. 165(PA), pages 840-852.
    17. Molina-Solana, Miguel & Ros, María & Ruiz, M. Dolores & Gómez-Romero, Juan & Martin-Bautista, M.J., 2017. "Data science for building energy management: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 70(C), pages 598-609.
    18. Jonathan Berrisch & Micha{l} Narajewski & Florian Ziel, 2022. "High-Resolution Peak Demand Estimation Using Generalized Additive Models and Deep Neural Networks," Papers 2203.03342, arXiv.org, revised Nov 2022.
    19. Alhamwi, Alaa & Medjroubi, Wided & Vogt, Thomas & Agert, Carsten, 2018. "Modelling urban energy requirements using open source data and models," Applied Energy, Elsevier, vol. 231(C), pages 1100-1108.
    20. Li, Wei & Lu, Can, 2019. "The multiple effectiveness of state natural gas consumption constraint policies for achieving sustainable development targets in China," Applied Energy, Elsevier, vol. 235(C), pages 685-698.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:264:y:2023:i:c:s036054422203095x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.