IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v304y2021ics0306261921010096.html
   My bibliography  Save this article

Data-driven district energy management with surrogate models and deep reinforcement learning

Author

Listed:
  • Pinto, Giuseppe
  • Deltetto, Davide
  • Capozzoli, Alfonso

Abstract

Demand side management at district scale plays a crucial role in the energy transition process, being an ideal candidate to balance the needs of both users and grid, by managing the volatility of renewable sources and increasing energy flexibility. The presented study aims to explore the benefits of a coordinated approach for the energy management of a cluster of buildings to optimise the electrical demand profiles and provide services to the grid without penalising indoor comfort conditions. The proposed methodology makes use of a fully data-driven control scheme which exploits Long Short-Term Memory (LSTM) Neural Networks, and Deep Reinforcement Learning (DRL). A simulation environment is introduced to train a DRL controller to manage the operation of heat pumps and chilled and domestic hot water storage for a cluster of four buildings. LSTM models are trained with synthetic data set created in EnergyPlus and are integrated into simulation environment to evaluate the indoor temperature dynamics in each building. The developed DRL controller is tested against a manually optimised Rule Based Controller (RBC). Results show that the DRL algorithm is able to reduce the overall cluster electricity costs, while decreasing the peak energy demand by 23% and the Peak to Average Ratio (PAR) by 20%, without penalizing indoor temperature control.

Suggested Citation

  • Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    • Handle: RePEc:eee:appene:v:304:y:2021:i:c:s0306261921010096
    DOI: 10.1016/j.apenergy.2021.117642
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921010096
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2021.117642?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kazmi, H. & D’Oca, S. & Delmastro, C. & Lodeweyckx, S. & Corgnati, S.P., 2016. "Generalizable occupant-driven optimization model for domestic hot water production in NZEB," Applied Energy, Elsevier, vol. 175(C), pages 1-15.
    2. Wang, Xiaodi & Liu, Youbo & Zhao, Junbo & Liu, Chang & Liu, Junyong & Yan, Jinyue, 2021. "Surrogate model enabled deep reinforcement learning for hybrid energy community operation," Applied Energy, Elsevier, vol. 289(C).
    3. Gianluca Serale & Massimo Fiorentini & Alfonso Capozzoli & Daniele Bernardini & Alberto Bemporad, 2018. "Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities," Energies, MDPI, vol. 11(3), pages 1-35, March.
    4. Auer, Hans & Haas, Reinhard, 2016. "On integrating large shares of variable renewables into the electricity system," Energy, Elsevier, vol. 115(P3), pages 1592-1601.
    5. Lu, Renzhi & Hong, Seung Ho, 2019. "Incentive-based demand response for smart grid with reinforcement learning and deep neural network," Applied Energy, Elsevier, vol. 236(C), pages 937-949.
    6. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    7. Ondeck, Abigail D. & Edgar, Thomas F. & Baldea, Michael, 2018. "Impact of rooftop photovoltaics and centralized energy storage on the design and operation of a residential CHP system," Applied Energy, Elsevier, vol. 222(C), pages 280-299.
    8. O'Shaughnessy, Eric & Cutler, Dylan & Ardani, Kristen & Margolis, Robert, 2018. "Solar plus: Optimization of distributed solar PV through battery storage and dispatchable load in residential buildings," Applied Energy, Elsevier, vol. 213(C), pages 11-21.
    9. Hussein Jumma Jabir & Jiashen Teh & Dahaman Ishak & Hamza Abunima, 2018. "Impacts of Demand-Side Management on Electrical Power Systems: A Review," Energies, MDPI, vol. 11(5), pages 1-19, April.
    10. Tang, Rui & Wang, Shengwei, 2019. "Model predictive control for thermal energy storage and thermal comfort optimization of building demand response in smart grids," Applied Energy, Elsevier, vol. 242(C), pages 873-882.
    11. Huang, Pei & Fan, Cheng & Zhang, Xingxing & Wang, Jiayuan, 2019. "A hierarchical coordinated demand response control for buildings with improved performances at building group," Applied Energy, Elsevier, vol. 242(C), pages 684-694.
    12. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    13. Celik, Berk & Roche, Robin & Suryanarayanan, Siddharth & Bouquain, David & Miraoui, Abdellatif, 2017. "Electric energy management in residential areas through coordination of multiple smart homes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 80(C), pages 260-275.
    14. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    15. Xiaodong Li & Maurice Clerc, 2019. "Correction to: Swarm Intelligence," International Series in Operations Research & Management Science, in: Michel Gendreau & Jean-Yves Potvin (ed.), Handbook of Metaheuristics, edition 3, chapter 19, pages C1-C1, Springer.
    16. Lund, Peter D. & Lindgren, Juuso & Mikkola, Jani & Salpakari, Jyri, 2015. "Review of energy system flexibility measures to enable high levels of variable renewable electricity," Renewable and Sustainable Energy Reviews, Elsevier, vol. 45(C), pages 785-807.
    17. Hui, Hongxun & Ding, Yi & Liu, Weidong & Lin, You & Song, Yonghua, 2017. "Operating reserve evaluation of aggregated air conditioners," Applied Energy, Elsevier, vol. 196(C), pages 218-228.
    18. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    19. Guerrero, Jaysson & Gebbran, Daniel & Mhanna, Sleiman & Chapman, Archie C. & Verbič, Gregor, 2020. "Towards a transactive energy system for integration of distributed energy resources: Home energy management, distributed optimal power flow, and peer-to-peer energy trading," Renewable and Sustainable Energy Reviews, Elsevier, vol. 132(C).
    20. Yang, Lei & Nagy, Zoltan & Goffin, Philippe & Schlueter, Arno, 2015. "Reinforcement learning for optimal control of low exergy buildings," Applied Energy, Elsevier, vol. 156(C), pages 577-586.
    21. Haider, Haider Tarish & See, Ong Hang & Elmenreich, Wilfried, 2016. "A review of residential demand response of smart grid," Renewable and Sustainable Energy Reviews, Elsevier, vol. 59(C), pages 166-178.
    22. Afroz, Zakia & Urmee, Tania & Shafiullah, G.M. & Higgins, Gary, 2018. "Real-time prediction model for indoor temperature in a commercial building," Applied Energy, Elsevier, vol. 231(C), pages 29-53.
    23. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    24. Xiaodong Li & Maurice Clerc, 2019. "Swarm Intelligence," International Series in Operations Research & Management Science, in: Michel Gendreau & Jean-Yves Potvin (ed.), Handbook of Metaheuristics, edition 3, chapter 0, pages 353-384, Springer.
    25. Wang, Andong & Li, Rongling & You, Shi, 2018. "Development of a data driven approach to explore the energy flexibility potential of building clusters," Applied Energy, Elsevier, vol. 232(C), pages 89-100.
    26. Verschae, Rodrigo & Kawashima, Hiroaki & Kato, Takekazu & Matsuyama, Takashi, 2016. "Coordinated energy management for inter-community imbalance minimization," Renewable Energy, Elsevier, vol. 87(P2), pages 922-935.
    27. Demary, Vera & Goecke, Henry, 2019. "Künstliche Intelligenz: Israel und Finnland vor China," IW-Kurzberichte 8/2019, Institut der deutschen Wirtschaft (IW) / German Economic Institute.
    28. Davide Coraci & Silvio Brandi & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Online Implementation of a Soft Actor-Critic Agent to Enhance Indoor Temperature Control and Energy Efficiency in Buildings," Energies, MDPI, vol. 14(4), pages 1-26, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    2. Song, Yuguang & Xia, Mingchao & Chen, Qifang & Chen, Fangjian, 2023. "A data-model fusion dispatch strategy for the building energy flexibility based on the digital twin," Applied Energy, Elsevier, vol. 332(C).
    3. Pasta, Edoardo & Faedo, Nicolás & Mattiazzo, Giuliana & Ringwood, John V., 2023. "Towards data-driven and data-based control of wave energy systems: Classification, overview, and critical assessment," Renewable and Sustainable Energy Reviews, Elsevier, vol. 188(C).
    4. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    5. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    6. Zhuang, Dian & Gan, Vincent J.L. & Duygu Tekler, Zeynep & Chong, Adrian & Tian, Shuai & Shi, Xing, 2023. "Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning," Applied Energy, Elsevier, vol. 338(C).
    7. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    8. Chen, Minghao & Sun, Yi & Xie, Zhiyuan & Lin, Nvgui & Wu, Peng, 2023. "An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning," Energy, Elsevier, vol. 284(C).
    9. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    10. Mu, Yunfei & Xu, Yurui & Cao, Yan & Chen, Wanqing & Jia, Hongjie & Yu, Xiaodan & Jin, Xiaolong, 2022. "A two-stage scheduling method for integrated community energy system based on a hybrid mechanism and data-driven model," Applied Energy, Elsevier, vol. 323(C).
    11. Luo, Na & Langevin, Jared & Chandra-Putra, Handi & Lee, Sang Hoon, 2022. "Quantifying the effect of multiple load flexibility strategies on commercial building electricity demand and services via surrogate modeling," Applied Energy, Elsevier, vol. 309(C).
    12. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    13. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    14. Charalampos Rafail Lazaridis & Iakovos Michailidis & Georgios Karatzinis & Panagiotis Michailidis & Elias Kosmatopoulos, 2024. "Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management," Energies, MDPI, vol. 17(3), pages 1-33, January.
    15. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    16. Byung-Ki Jeon & Eui-Jong Kim, 2022. "White-Model Predictive Control for Balancing Energy Savings and Thermal Comfort," Energies, MDPI, vol. 15(7), pages 1-12, March.
    17. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    2. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    5. Charbonnier, Flora & Morstyn, Thomas & McCulloch, Malcolm D., 2022. "Coordination of resources at the edge of the electricity grid: Systematic review and taxonomy," Applied Energy, Elsevier, vol. 318(C).
    6. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    7. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    8. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    9. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.
    10. Han, Gwangwoo & Joo, Hong-Jin & Lim, Hee-Won & An, Young-Sub & Lee, Wang-Je & Lee, Kyoung-Ho, 2023. "Data-driven heat pump operation strategy using rainbow deep reinforcement learning for significant reduction of electricity cost," Energy, Elsevier, vol. 270(C).
    11. Kathirgamanathan, Anjukan & De Rosa, Mattia & Mangina, Eleni & Finn, Donal P., 2021. "Data-driven predictive control for unlocking building energy flexibility: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 135(C).
    12. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).
    13. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    14. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    15. Ussama Assad & Muhammad Arshad Shehzad Hassan & Umar Farooq & Asif Kabir & Muhammad Zeeshan Khan & S. Sabahat H. Bukhari & Zain ul Abidin Jaffri & Judit Oláh & József Popp, 2022. "Smart Grid, Demand Response and Optimization: A Critical Review of Computational Methods," Energies, MDPI, vol. 15(6), pages 1-36, March.
    16. Wen, Lulu & Zhou, Kaile & Li, Jun & Wang, Shanyong, 2020. "Modified deep learning and reinforcement learning for an incentive-based demand response model," Energy, Elsevier, vol. 205(C).
    17. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    18. Antonopoulos, Ioannis & Robu, Valentin & Couraud, Benoit & Kirli, Desen & Norbu, Sonam & Kiprakis, Aristides & Flynn, David & Elizondo-Gonzalez, Sergio & Wattam, Steve, 2020. "Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 130(C).
    19. Zahra Fallahi & Gregor P. Henze, 2019. "Interactive Buildings: A Review," Sustainability, MDPI, vol. 11(14), pages 1-26, July.
    20. Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:304:y:2021:i:c:s0306261921010096. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.