IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v313y2024ics0360544224034832.html
   My bibliography  Save this article

Incremental learning user profile and deep reinforcement learning for managing building energy in heating water

Author

Listed:
  • Yin, Linfei
  • Xiong, Yi

Abstract

Deep reinforcement learning (DRL) has garnered growing attention as a data-driven control technique in the field of built environments. However, the existing DRL approaches for managing water systems cannot consider information from multiple time steps, are prone to overestimation, fall into the problem of locally optimal solutions, and fail to cope with time-varying environments, resulting in an inability to minimize energy consumption while considering water comfort and hygiene of occupants. Therefore, this study proposes an incremental learning user profile and deep reinforcement learning (ILUPDRL) method for controlling hot water systems. This study employs hot water user profiles to reflect the hot water demand (HWD) habits. The proposed ILUPDRL addresses the challenges arising from evolving HWD through incremental learning of hot water user profiles. Moreover, to enable the ILUPDRL to consider information from multiple time steps, this study proposes the recurrent proximal policy optimization (RPPO) algorithm and integrates the RPPO into the ILUPDRL. The simulation results show that the ILUPDRL achieves up to 67.53 % energy savings while considering the water comfort and water hygiene of occupants.

Suggested Citation

  • Yin, Linfei & Xiong, Yi, 2024. "Incremental learning user profile and deep reinforcement learning for managing building energy in heating water," Energy, Elsevier, vol. 313(C).
    • Handle: RePEc:eee:energy:v:313:y:2024:i:c:s0360544224034832
    DOI: 10.1016/j.energy.2024.133705
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544224034832
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2024.133705?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fang, Lei & He, Bin, 2023. "A deep learning framework using multi-feature fusion recurrent neural networks for energy consumption forecasting," Applied Energy, Elsevier, vol. 348(C).
    2. Ng, Rong Wang & Begam, Kasim Mumtaj & Rajkumar, Rajprasad Kumar & Wong, Yee Wan & Chong, Lee Wai, 2021. "An improved self-organizing incremental neural network model for short-term time-series load prediction," Applied Energy, Elsevier, vol. 292(C).
    3. Li, Wei & Markides, Christos N. & Zeng, Min & Peng, Jian, 2024. "4E evaluations of salt hydrate-based solar thermochemical heat transformer system used for domestic hot water production," Energy, Elsevier, vol. 286(C).
    4. Langer, Lissy & Volling, Thomas, 2022. "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 327(C).
    5. Rouleau, Jean & Gosselin, Louis, 2021. "Impacts of the COVID-19 lockdown on energy consumption in a Canadian social housing building," Applied Energy, Elsevier, vol. 287(C).
    6. Gursoy, Mehmet & Dincer, Ibrahim, 2023. "A solar based system for integrated production of power, heat, hot water and cooling," Energy, Elsevier, vol. 282(C).
    7. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    8. Lee, Jae Yong & Yim, Taesu, 2021. "Energy and flow demand analysis of domestic hot water in an apartment complex using a smart meter," Energy, Elsevier, vol. 229(C).
    9. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
    10. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    11. Kim, Dongwoo & Yim, Taesu & Lee, Jae Yong, 2021. "Analytical study on changes in domestic hot water use caused by COVID-19 pandemic," Energy, Elsevier, vol. 231(C).
    12. Xu, Zhongwen & Tan, Shiqi & Yao, Liming & Lv, Chengwei, 2024. "Exploring water-saving potentials of US electric power transition while thirsting for carbon neutrality," Energy, Elsevier, vol. 292(C).
    13. Clift, Dean Holland & Stanley, Cameron & Hasan, Kazi N. & Rosengarten, Gary, 2023. "Assessment of advanced demand response value streams for water heaters in renewable-rich electricity markets," Energy, Elsevier, vol. 267(C).
    14. Qu, Minglu & Yan, Xufeng & Wang, Haiyang & Hei, Yingxiao & Liu, Hongzhi & Li, Zhao, 2022. "Energy, exergy, economic and environmental analysis of photovoltaic/thermal integrated water source heat pump water heater," Renewable Energy, Elsevier, vol. 194(C), pages 1084-1097.
    15. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    16. Wei, Wu & Skye, Harrison M., 2021. "Residential net-zero energy buildings: Review and perspective," Renewable and Sustainable Energy Reviews, Elsevier, vol. 142(C).
    17. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach," Applied Energy, Elsevier, vol. 312(C).
    18. Nakıp, Mert & Çopur, Onur & Biyik, Emrah & Güzeliş, Cüneyt, 2023. "Renewable energy management in smart home environment via forecast embedded scheduling based on Recurrent Trend Predictive Neural Network," Applied Energy, Elsevier, vol. 340(C).
    19. Puah, Boon Keat & Chong, Lee Wai & Wong, Yee Wan & Begam, K.M. & Khan, Nafizah & Juman, Mohammed Ayoub & Rajkumar, Rajprasad Kumar, 2021. "A regression unsupervised incremental learning algorithm for solar irradiance prediction," Renewable Energy, Elsevier, vol. 164(C), pages 908-925.
    20. Zhou, Xin & Tian, Shuai & An, Jingjing & Yan, Da & Zhang, Lun & Yang, Junyan, 2022. "Modeling occupant behavior’s influence on the energy efficiency of solar domestic hot water systems," Applied Energy, Elsevier, vol. 309(C).
    21. Zhang, Weiyi & Zhou, Haiyang & Bao, Xiaohua & Cui, Hongzhi, 2023. "Outlet water temperature prediction of energy pile based on spatial-temporal feature extraction through CNN–LSTM hybrid model," Energy, Elsevier, vol. 264(C).
    22. Zhang, Zhen & Zhang, Tiezhu & Hong, Jichao & Zhang, Hongxin & Yang, Jian & Jia, Qingxiao, 2023. "Double deep Q-network guided energy management strategy of a novel electric-hydraulic hybrid electric vehicle," Energy, Elsevier, vol. 269(C).
    23. Li, Yang & Bu, Fanjin & Li, Yuanzheng & Long, Chao, 2023. "Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 333(C).
    24. Zhao, Jing & Guo, Yiyi & Lin, Yihua & Zhao, Zhiyuan & Guo, Zhenhai, 2024. "A novel dynamic ensemble of numerical weather prediction for multi-step wind speed forecasting with deep reinforcement learning and error sequence modeling," Energy, Elsevier, vol. 302(C).
    25. Kudela, Libor & Špiláček, Michal & Pospíšil, Jiří, 2021. "Influence of control strategy on seasonal coefficient of performance for a heat pump with low-temperature heat storage in the geographical conditions of Central Europe," Energy, Elsevier, vol. 234(C).
    26. Xiao, Boyi & Yang, Weiwei & Wu, Jiamin & Walker, Paul D. & Zhang, Nong, 2022. "Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle," Energy, Elsevier, vol. 253(C).
    27. Na, Wei & Wang, Mingming, 2022. "A Bayesian approach with urban-scale energy model to calibrate building energy consumption for space heating: A case study of application in Beijing," Energy, Elsevier, vol. 247(C).
    28. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hall, Rebecca & Kenway, Steven & O'Brien, Katherine & Memon, Fayyaz, 2025. "Quantification of residential water-related energy needs cohesion, validation and global representation to unlock efficiency gains," Renewable and Sustainable Energy Reviews, Elsevier, vol. 207(C).
    2. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Heidari, Amirreza & Girardin, Luc & Dorsaz, Cédric & Maréchal, François, 2025. "A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation," Applied Energy, Elsevier, vol. 378(PA).
    5. Panagiotis Michailidis & Iakovos Michailidis & Elias Kosmatopoulos, 2025. "Reinforcement Learning for Optimizing Renewable Energy Utilization in Buildings: A Review on Applications and Innovations," Energies, MDPI, vol. 18(7), pages 1-40, March.
    6. Zhang, Hao & Lei, Nuo & Chen, Boli & Li, Bingbing & Li, Rulong & Wang, Zhi, 2024. "Modeling and control system optimization for electrified vehicles: A data-driven approach," Energy, Elsevier, vol. 310(C).
    7. Łukasz Amanowicz, 2021. "Peak Power of Heat Source for Domestic Hot Water Preparation (DHW) for Residential Estate in Poland as a Representative Case Study for the Climate of Central Europe," Energies, MDPI, vol. 14(23), pages 1-15, December.
    8. Indre Siksnelyte-Butkiene, 2021. "Impact of the COVID-19 Pandemic to the Sustainability of the Energy Sector," Sustainability, MDPI, vol. 13(23), pages 1-19, November.
    9. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
    10. Mudhafar Al-Saadi & Maher Al-Greer & Michael Short, 2023. "Reinforcement Learning-Based Intelligent Control Strategies for Optimal Power Management in Advanced Power Distribution Systems: A Survey," Energies, MDPI, vol. 16(4), pages 1-38, February.
    11. Ibrahim Ali Kachalla & Christian Ghiaus, 2024. "Electric Water Boiler Energy Prediction: State-of-the-Art Review of Influencing Factors, Techniques, and Future Directions," Energies, MDPI, vol. 17(2), pages 1-32, January.
    12. Schmitz, Simon & Brucke, Karoline & Kasturi, Pranay & Ansari, Esmail & Klement, Peter, 2024. "Forecast-based and data-driven reinforcement learning for residential heat pump operation," Applied Energy, Elsevier, vol. 371(C).
    13. Zheng, Zhihang & Jin, Yipeng & Zhou, Jin & Yang, Ying & Xu, Feng & Liu, Hongcheng, 2025. "A novel dynamic operation method for solar assisted air source heat pump systems: Optimization control and performance analysis," Energy, Elsevier, vol. 316(C).
    14. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    15. He, Hongwen & Su, Qicong & Huang, Ruchen & Niu, Zegong, 2024. "Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm," Energy, Elsevier, vol. 294(C).
    16. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    17. Chen, Jie & Wang, Ruochen & Ding, Renkai & Luo, Ding, 2024. "Matching design and numerical optimization of automotive thermoelectric generator system applied to range-extended electric vehicle," Applied Energy, Elsevier, vol. 370(C).
    18. Despoina Antypa & Foteini Petrakli & Anastasia Gkika & Pamela Voigt & Alexander Kahnt & Robert Böhm & Jan Suchorzewski & Andreia Araújo & Susana Sousa & Elias P. Koumoulos, 2022. "Life Cycle Assessment of Advanced Building Components towards NZEBs," Sustainability, MDPI, vol. 14(23), pages 1-20, December.
    19. Gao, Xianhui & Wang, Sheng & Sun, Ying & Zhai, Junyi & Chen, Nan & Zhang, Xiao-Ping, 2024. "Low-carbon energy scheduling for integrated energy systems considering offshore wind power hydrogen production and dynamic hydrogen doping strategy," Applied Energy, Elsevier, vol. 376(PA).
    20. Haleh Moghaddasi & Charles Culp & Jorge Vanegas & Saptarshi Das & Mehrdad Ehsani, 2022. "An Adaptable Net Zero Model: Energy Analysis of a Monitored Case Study," Energies, MDPI, vol. 15(11), pages 1-24, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:313:y:2024:i:c:s0360544224034832. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.