IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v337y2025ics0360544225041593.html

Price-responsive control using deep reinforcement learning for heating systems: Simulation and living lab experiment

Author

Listed:
  • Mokhtari, Reza
  • Montazeri, Mina
  • Cai, Hanmin
  • Heer, Philipp
  • Li, Rongling

Abstract

While buildings are one of the main sources of global carbon emissions, they also present significant opportunities for decarbonization. Building energy systems are typically controlled using pre-defined schedules and rule-based strategies, which usually lack adaptability to dynamic conditions such as changing weather states, occupancy and energy prices. In this study, we develop a price-responsive Deep Reinforcement Learning (DRL) controller to autonomously control the indoor temperature of a building. The designed controller aims to minimize heating costs while keeping the indoor temperature within the acceptable comfort bounds, responding to dynamic energy prices. To improve learning efficiency, this study introduces Temporally-Weighted State Difference (TSD), a method to compress future price data into a single state observation for the controller, enabling future-aware decisions while reducing training complexity. The controller is developed and trained in a simulation environment using a physically consistent neural network model. It is then implemented in a real-world residential unit in the NEST building in Dübendorf, Switzerland, and evaluated during the winter of 2024 to 2025 using the dynamic electricity prices of the Swiss market. Experimental results indicate that incorporating future price information leads to more informed decisions by the controller and improves the performance of DRL by 11.5%. Experimental results show a 79% reduction in heating costs compared to a rule-based controller, at a cost of exceeding the thermal comfort range by 0.3 °C on average. The study demonstrates the potential of price-responsive DRL in reducing energy costs and providing demand-side flexibility to the energy grid.

Suggested Citation

  • Mokhtari, Reza & Montazeri, Mina & Cai, Hanmin & Heer, Philipp & Li, Rongling, 2025. "Price-responsive control using deep reinforcement learning for heating systems: Simulation and living lab experiment," Energy, Elsevier, vol. 337(C).
    • Handle: RePEc:eee:energy:v:337:y:2025:i:c:s0360544225041593
    DOI: 10.1016/j.energy.2025.138517
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544225041593
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2025.138517?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Kazmi, Hussain & Suykens, Johan & Balint, Attila & Driesen, Johan, 2019. "Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads," Applied Energy, Elsevier, vol. 238(C), pages 1022-1035.
    2. Tina, Giuseppe Marco & Aneli, Stefano & Gagliano, Antonio, 2022. "Technical and economic analysis of the provision of ancillary services through the flexibility of HVAC system in shopping centers," Energy, Elsevier, vol. 258(C).
    3. Montazeri, Mina & Remlinger, Carl & Bejar Haro, Benjamin & Heer, Philipp, 2025. "Fully data-driven and modular building thermal control with physically consistent modeling," Applied Energy, Elsevier, vol. 390(C).
    4. Mokhtari, Reza & Junker, Rune Grønborg & Madsen, Henrik & Li, Rongling, 2025. "A methodological framework for designing dynamic heat price for demand response in district heating," Energy, Elsevier, vol. 324(C).
    5. Guo, Fangzhou & Ham, Sang woo & Kim, Donghun & Moon, Hyeun Jun, 2025. "Deep reinforcement learning control for co-optimizing energy consumption, thermal comfort, and indoor air quality in an office building," Applied Energy, Elsevier, vol. 377(PA).
    6. Carlo Carraro & Alessandro Lanza & Massimo Tavoni, 2014. "All You Need to Know About the IPCC 5th Assessment Report. Mitigation of Climate Change," Review of Environment, Energy and Economics - Re3, Fondazione Eni Enrico Mattei, April.
    7. Di Natale, L. & Svetozarevic, B. & Heer, P. & Jones, C.N., 2022. "Physically Consistent Neural Networks for building thermal modeling: Theory and analysis," Applied Energy, Elsevier, vol. 325(C).
    8. Fu, Yangyang & Xu, Shichao & Zhu, Qi & O’Neill, Zheng & Adetola, Veronica, 2023. "How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system," Energy, Elsevier, vol. 273(C).
    9. Svetozarevic, B. & Baumann, C. & Muntwiler, S. & Di Natale, L. & Zeilinger, M.N. & Heer, P., 2022. "Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments," Applied Energy, Elsevier, vol. 307(C).
    10. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    11. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    12. Wang, Xiangwei & Wang, Peng & Huang, Renke & Zhu, Xiuli & Arroyo, Javier & Li, Ning, 2025. "Safe deep reinforcement learning for building energy management," Applied Energy, Elsevier, vol. 377(PA).
    13. Liu, Xiangfei & Ren, Mifeng & Yang, Zhile & Yan, Gaowei & Guo, Yuanjun & Cheng, Lan & Wu, Chengke, 2022. "A multi-step predictive deep reinforcement learning algorithm for HVAC control systems in smart buildings," Energy, Elsevier, vol. 259(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Yu, Peipei & Zhang, Hongcai & Song, Yonghua & Wang, Zhenyi & Dong, Huiyu & Ji, Liang, 2025. "Safe reinforcement learning for power system control: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 223(C).
    3. Panagiotis Michailidis & Iakovos Michailidis & Elias Kosmatopoulos, 2025. "Reinforcement Learning for Electric Vehicle Charging Management: Theory and Applications," Energies, MDPI, vol. 18(19), pages 1-50, October.
    4. Chen, Siliang & Liang, Xinbin & Liu, Ying & Li, Xilin & Jin, Xinqiao & Du, Zhimin, 2025. "Customized large-scale model for human-AI collaborative operation and maintenance management of building energy systems," Applied Energy, Elsevier, vol. 393(C).
    5. Di Natale, L. & Svetozarevic, B. & Heer, P. & Jones, C.N., 2023. "Towards scalable physically consistent neural networks: An application to data-driven multi-zone thermal building models," Applied Energy, Elsevier, vol. 340(C).
    6. Ajagekar, Akshay & Decardi-Nelson, Benjamin & You, Fengqi, 2024. "Energy management for demand response in networked greenhouses with multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 355(C).
    7. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    8. Pavirani, Fabio & Van Gompel, Jonas & Karimi Madahi, Seyed Soroush & Claessens, Bert & Develder, Chris, 2025. "Predicting and publishing accurate imbalance prices using Monte Carlo Tree Search," Applied Energy, Elsevier, vol. 392(C).
    9. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    10. Xiao, Tianqi & You, Fengqi, 2024. "Physically consistent deep learning-based day-ahead energy dispatching and thermal comfort control for grid-interactive communities," Applied Energy, Elsevier, vol. 353(PB).
    11. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    12. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    13. Christian Blad & Simon Bøgh & Carsten Kallesøe, 2021. "A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems," Energies, MDPI, vol. 14(22), pages 1-20, November.
    14. Zeng, Lanting & Qiu, Dawei & Sun, Mingyang, 2022. "Resilience enhancement of multi-agent reinforcement learning-based demand response against adversarial attacks," Applied Energy, Elsevier, vol. 324(C).
    15. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    16. Jiang, Fuyang & Kazmi, Hussain, 2025. "What-if: A causal machine learning approach to control-oriented modelling for building thermal dynamics," Applied Energy, Elsevier, vol. 377(PC).
    17. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    18. Xu, Wenjie & Svetozarevic, Bratislav & Di Natale, Loris & Heer, Philipp & Jones, Colin N., 2024. "Data-driven adaptive building thermal controller tuning with constraints: A primal–dual contextual Bayesian optimization approach," Applied Energy, Elsevier, vol. 358(C).
    19. Muhammad Ikram & Daryoush Habibi & Asma Aziz, 2025. "Networked Multi-Agent Deep Reinforcement Learning Framework for the Provision of Ancillary Services in Hybrid Power Plants," Energies, MDPI, vol. 18(10), pages 1-34, May.
    20. Nik, Vahid M. & Hosseini, Mohammad, 2023. "CIRLEM: a synergic integration of Collective Intelligence and Reinforcement learning in Energy Management for enhanced climate resilience and lightweight computation," Applied Energy, Elsevier, vol. 350(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:337:y:2025:i:c:s0360544225041593. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.