IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v307y2022ics0306261921014045.html
   My bibliography  Save this article

Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments

Author

Listed:
  • Svetozarevic, B.
  • Baumann, C.
  • Muntwiler, S.
  • Di Natale, L.
  • Zeilinger, M.N.
  • Heer, P.

Abstract

The control of modern buildings is a complex multi-loop problem due to the integration of renewable energy generation, storage devices, and electric vehicles (EVs). Additionally, it is a complex multi-criteria problem due to the need to optimize overall energy use while satisfying users’ comfort. Both conventional rule-based (RB) controllers, which are difficult to apply in multi-loop settings, and advanced model-based controllers, which require an accurate building model, cannot fulfil the requirements of the building automation industry to solve this problem optimally at low development and commissioning costs. This work presents a fully data-driven pipeline to obtain an optimal control policy from historical building and weather data, thus avoiding the need for complex physics-based modelling. We demonstrate the potential of this method by jointly controlling a room temperature and an EV to minimize the cost of electricity while retaining the comfort of the occupants. We model the room temperature with a recurrent neural network and use it as a simulation environment to learn a deep reinforcement learning (DRL) control policy. It achieves on average 17% energy savings and 19% better comfort satisfaction than a standard RB room temperature controller. When a bidirectional EV is connected additionally and a two-tariff electricity pricing is applied, it successfully leverages the battery and decreases the overall cost of electricity. Finally, we deployed it on a real building, where it achieved up to 30% energy savings while maintaining similar comfort levels compared to a conventional RB room temperature controller.

Suggested Citation

  • Svetozarevic, B. & Baumann, C. & Muntwiler, S. & Di Natale, L. & Zeilinger, M.N. & Heer, P., 2022. "Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments," Applied Energy, Elsevier, vol. 307(C).
  • Handle: RePEc:eee:appene:v:307:y:2022:i:c:s0306261921014045
    DOI: 10.1016/j.apenergy.2021.118127
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921014045
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2021.118127?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhou, Yuekuan & Cao, Sunliang & Hensen, Jan L.M. & Lund, Peter D., 2019. "Energy integration and interaction between buildings and vehicles: A state-of-the-art review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 114(C), pages 1-1.
    2. Park, June Young & Nagy, Zoltan, 2018. "Comprehensive analysis of the relationship between thermal comfort and building control research - A data-driven literature review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 82(P3), pages 2664-2679.
    3. Shaikh, Pervez Hameed & Nor, Nursyarizal Bin Mohd & Nallagownden, Perumal & Elamvazuthi, Irraivan & Ibrahim, Taib, 2014. "A review on optimized control systems for building energy and comfort management of smart sustainable buildings," Renewable and Sustainable Energy Reviews, Elsevier, vol. 34(C), pages 409-429.
    4. Smarra, Francesco & Jain, Achin & de Rubeis, Tullio & Ambrosini, Dario & D’Innocenzo, Alessandro & Mangharam, Rahul, 2018. "Data-driven model predictive control using random forests for building energy optimization and climate control," Applied Energy, Elsevier, vol. 226(C), pages 1252-1272.
    5. Gianluca Serale & Massimo Fiorentini & Alfonso Capozzoli & Daniele Bernardini & Alberto Bemporad, 2018. "Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities," Energies, MDPI, vol. 11(3), pages 1-35, March.
    6. Sunyong Kim & Hyuk Lim, 2018. "Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings," Energies, MDPI, vol. 11(8), pages 1-19, August.
    7. Huang, Pei & Lovati, Marco & Zhang, Xingxing & Bales, Chris, 2020. "A coordinated control to improve performance for a building cluster with energy storage, electric vehicles, and energy sharing considered," Applied Energy, Elsevier, vol. 268(C).
    8. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    9. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    10. Verhelst J., & Van Ham G., & Saelens D., & Helsen L.,, 2017. "Model selection for continuous commissioning of HVAC-systems in office buildings: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 76(C), pages 673-686.
    11. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    12. David Silver & Julian Schrittwieser & Karen Simonyan & Ioannis Antonoglou & Aja Huang & Arthur Guez & Thomas Hubert & Lucas Baker & Matthew Lai & Adrian Bolton & Yutian Chen & Timothy Lillicrap & Fan , 2017. "Mastering the game of Go without human knowledge," Nature, Nature, vol. 550(7676), pages 354-359, October.
    13. Chwieduk, Dorota, 2003. "Towards sustainable-energy buildings," Applied Energy, Elsevier, vol. 76(1-3), pages 211-217, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Omar al-Ani & Sanjoy Das & Hongyu Wu, 2023. "Imitation Learning with Deep Attentive Tabular Neural Networks for Environmental Prediction and Control in Smart Home," Energies, MDPI, vol. 16(13), pages 1-19, June.
    2. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    3. Ali, Aliyuda & Aliyuda, Kachalla & Elmitwally, Nouh & Muhammad Bello, Abdulwahab, 2022. "Towards more accurate and explainable supervised learning-based prediction of deliverability for underground natural gas storage," Applied Energy, Elsevier, vol. 327(C).
    4. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    5. Han, Dongho & Kwon, Sanguk & Lee, Miyoung & Kim, Jonghoon & Yoo, Kisoo, 2023. "Electrochemical impedance spectroscopy image transformation-based convolutional neural network for diagnosis of external environment classification affecting abnormal aging of Li-ion batteries," Applied Energy, Elsevier, vol. 345(C).
    6. Afaq Ahmad & Muhammad Khalid & Zahid Ullah & Naveed Ahmad & Mohammad Aljaidi & Faheem Ahmed Malik & Umar Manzoor, 2022. "Electric Vehicle Charging Modes, Technologies and Applications of Smart Charging," Energies, MDPI, vol. 15(24), pages 1-32, December.
    7. Di Natale, L. & Svetozarevic, B. & Heer, P. & Jones, C.N., 2023. "Towards scalable physically consistent neural networks: An application to data-driven multi-zone thermal building models," Applied Energy, Elsevier, vol. 340(C).
    8. Maria Ferrara & Matteo Bilardo & Dragos-Ioan Bogatu & Doyun Lee & Mahmood Khatibi & Samira Rahnama & Jun Shinoda & Ying Sun & Yongjun Sun & Alireza Afshari & Fariborz Haghighat & Ongun B. Kazanci & Ry, 2024. "Review on Advanced Storage Control Applied to Optimized Operation of Energy Systems for Buildings and Districts: Insights and Perspectives," Energies, MDPI, vol. 17(14), pages 1-26, July.
    9. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    10. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    11. Xu, Wenjie & Svetozarevic, Bratislav & Di Natale, Loris & Heer, Philipp & Jones, Colin N., 2024. "Data-driven adaptive building thermal controller tuning with constraints: A primal–dual contextual Bayesian optimization approach," Applied Energy, Elsevier, vol. 358(C).
    12. Hua Jin & Gang Meng & Yuanzhi Pan & Xing Zhang & Changda Wang, 2022. "An Improved Intelligent Control System for Temperature and Humidity in a Pig House," Agriculture, MDPI, vol. 12(12), pages 1-21, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Touzani, Samir & Prakash, Anand Krishnan & Wang, Zhe & Agarwal, Shreya & Pritoni, Marco & Kiran, Mariam & Brown, Richard & Granderson, Jessica, 2021. "Controlling distributed energy resources via deep reinforcement learning for load flexibility and energy efficiency," Applied Energy, Elsevier, vol. 304(C).
    3. Clara Ceccolini & Roozbeh Sangi, 2022. "Benchmarking Approaches for Assessing the Performance of Building Control Strategies: A Review," Energies, MDPI, vol. 15(4), pages 1-30, February.
    4. Elsisi, Mahmoud & Amer, Mohammed & Dababat, Alya’ & Su, Chun-Lien, 2023. "A comprehensive review of machine learning and IoT solutions for demand side energy management, conservation, and resilient operation," Energy, Elsevier, vol. 281(C).
    5. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    6. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Operational optimization for off-grid renewable building energy system using deep reinforcement learning," Applied Energy, Elsevier, vol. 325(C).
    7. Bünning, Felix & Huber, Benjamin & Schalbetter, Adrian & Aboudonia, Ahmed & Hudoba de Badyn, Mathias & Heer, Philipp & Smith, Roy S. & Lygeros, John, 2022. "Physics-informed linear regression is competitive with two Machine Learning methods in residential building MPC," Applied Energy, Elsevier, vol. 310(C).
    8. Kathirgamanathan, Anjukan & De Rosa, Mattia & Mangina, Eleni & Finn, Donal P., 2021. "Data-driven predictive control for unlocking building energy flexibility: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 135(C).
    9. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    10. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    11. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    12. Antonopoulos, Ioannis & Robu, Valentin & Couraud, Benoit & Kirli, Desen & Norbu, Sonam & Kiprakis, Aristides & Flynn, David & Elizondo-Gonzalez, Sergio & Wattam, Steve, 2020. "Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 130(C).
    13. Yang, Shiyu & Wan, Man Pun, 2022. "Machine-learning-based model predictive control with instantaneous linearization – A case study on an air-conditioning and mechanical ventilation system," Applied Energy, Elsevier, vol. 306(PB).
    14. Lilia Tightiz & Joon Yoo, 2022. "A Review on a Data-Driven Microgrid Management System Integrating an Active Distribution Network: Challenges, Issues, and New Trends," Energies, MDPI, vol. 15(22), pages 1-24, November.
    15. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    16. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    17. Han, Gwangwoo & Joo, Hong-Jin & Lim, Hee-Won & An, Young-Sub & Lee, Wang-Je & Lee, Kyoung-Ho, 2023. "Data-driven heat pump operation strategy using rainbow deep reinforcement learning for significant reduction of electricity cost," Energy, Elsevier, vol. 270(C).
    18. Lei, Yue & Zhan, Sicheng & Ono, Eikichi & Peng, Yuzhen & Zhang, Zhiang & Hasama, Takamasa & Chong, Adrian, 2022. "A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings," Applied Energy, Elsevier, vol. 324(C).
    19. Zhan, Sicheng & Chong, Adrian, 2021. "Data requirements and performance evaluation of model predictive control in buildings: A modeling perspective," Renewable and Sustainable Energy Reviews, Elsevier, vol. 142(C).
    20. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:307:y:2022:i:c:s0306261921014045. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.