IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v15y2022i19p7381-d936094.html
   My bibliography  Save this article

Energy Management Simulation with Multi-Agent Reinforcement Learning: An Approach to Achieve Reliability and Resilience

Author

Listed:
  • Kapil Deshpande

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria
    Current address: Profactor GmbH, Im Stadtgut D1 Steyr-Gleink, Upper Austria, 4407 Steyr, Austria.)

  • Philipp Möhl

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria)

  • Alexander Hämmerle

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria)

  • Georg Weichhart

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria)

  • Helmut Zörrer

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria)

  • Andreas Pichler

    (Profactor GmbH, Robotics and Automation Systems Department, 4407 Steyr, Austria)

Abstract

The share of energy produced by small-scale renewable energy sources, including photovoltaic panels and wind turbines, will significantly increase in the near future. These systems will be integrated in microgrids to strengthen the independence of energy consumers. This work deals with energy management in microgrids, taking into account the volatile nature of renewable energy sources. In the developed approach, Multi-Agent Reinforcement Learning is applied, where agents represent microgrid components. The individual agents are trained to make good decisions with respect to adapting to the energy load in the grid. Training of agents leverages the historic energy profile data for energy consumption and renewable energy production. The implemented energy management simulation shows good performance and balances the energy flows. The quantitative performance evaluation includes comparisons with the exact solutions from a linear program. The computational results demonstrate good generalisation capabilities of the trained agents and the impact of these capabilities on the reliability and resilience of energy management in microgrids.

Suggested Citation

  • Kapil Deshpande & Philipp Möhl & Alexander Hämmerle & Georg Weichhart & Helmut Zörrer & Andreas Pichler, 2022. "Energy Management Simulation with Multi-Agent Reinforcement Learning: An Approach to Achieve Reliability and Resilience," Energies, MDPI, vol. 15(19), pages 1-35, October.
    • Handle: RePEc:gam:jeners:v:15:y:2022:i:19:p:7381-:d:936094
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/15/19/7381/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/15/19/7381/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Cui, Zhenhua & Kang, Le & Li, Liwei & Wang, Licheng & Wang, Kai, 2022. "A combined state-of-charge estimation method for lithium-ion battery using an improved BGRU network and UKF," Energy, Elsevier, vol. 259(C).
    2. Hussain, Akhtar & Bui, Van-Hai & Kim, Hak-Man, 2019. "Microgrids as a resilience resource and strategies used by microgrids for enhancing resilience," Applied Energy, Elsevier, vol. 240(C), pages 56-72.
    3. Yu-Fang Wang, 2020. "Adaptive job shop scheduling strategy based on weighted Q-learning algorithm," Journal of Intelligent Manufacturing, Springer, vol. 31(2), pages 417-432, February.
    4. Khawaja Haider Ali & Marvin Sigalo & Saptarshi Das & Enrico Anderlini & Asif Ali Tahir & Mohammad Abusara, 2021. "Reinforcement Learning for Energy-Storage Systems in Grid-Connected Microgrids: An Investigation of Online vs. Offline Implementation," Energies, MDPI, vol. 14(18), pages 1-18, September.
    5. Dehghani, Nariman L. & Jeddi, Ashkan B. & Shafieezadeh, Abdollah, 2021. "Intelligent hurricane resilience enhancement of power distribution systems via deep reinforcement learning," Applied Energy, Elsevier, vol. 285(C).
    6. Xiaohan Fang & Jinkuan Wang & Guanru Song & Yinghua Han & Qiang Zhao & Zhiao Cao, 2019. "Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling," Energies, MDPI, vol. 13(1), pages 1-26, December.
    7. Gaurav Chaudhary & Jacob J. Lamb & Odne S. Burheim & Bjørn Austbø, 2021. "Review of Energy Storage and Energy Management System Control Strategies in Microgrids," Energies, MDPI, vol. 14(16), pages 1-26, August.
    8. Ying Ji & Jianhui Wang & Jiacan Xu & Xiaoke Fang & Huaguang Zhang, 2019. "Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning," Energies, MDPI, vol. 12(12), pages 1-21, June.
    9. A. T. D. Perera & Vahid M. Nik & Deliang Chen & Jean-Louis Scartezzini & Tianzhen Hong, 2020. "Quantifying the impacts of climate change and extreme climate events on energy systems," Nature Energy, Nature, vol. 5(2), pages 150-159, February.
    10. Grace Muriithi & Sunetra Chowdhury, 2021. "Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach," Energies, MDPI, vol. 14(9), pages 1-24, May.
    11. Lucas Cuadra & Sancho Salcedo-Sanz & Javier Del Ser & Silvia Jiménez-Fernández & Zong Woo Geem, 2015. "A Critical Review of Robustness in Power Grids Using Complex Networks Concepts," Energies, MDPI, vol. 8(9), pages 1-55, August.
    12. Jufri, Fauzan Hanif & Widiputra, Victor & Jung, Jaesung, 2019. "State-of-the-art review on power grid resilience to extreme weather events: Definitions, frameworks, quantitative assessment methodologies, and enhancement strategies," Applied Energy, Elsevier, vol. 239(C), pages 1049-1065.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sana Qaiyum & Martin Margala & Pravin R. Kshirsagar & Prasun Chakrabarti & Kashif Irshad, 2023. "Energy Performance Analysis of Photovoltaic Integrated with Microgrid Data Analysis Using Deep Learning Feature Selection and Classification Techniques," Sustainability, MDPI, vol. 15(14), pages 1-21, July.
    2. Mudhafar Al-Saadi & Maher Al-Greer & Michael Short, 2023. "Reinforcement Learning-Based Intelligent Control Strategies for Optimal Power Management in Advanced Power Distribution Systems: A Survey," Energies, MDPI, vol. 16(4), pages 1-38, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Hanchen & Wang, Chong & Ju, Ping & Li, Hongyu, 2022. "A sequentially preventive model enhancing power system resilience against extreme-weather-triggered failures," Renewable and Sustainable Energy Reviews, Elsevier, vol. 156(C).
    2. Wang, Zhaoqi & Zhang, Lu & Tang, Wei & Chen, Ying & Shen, Chen, 2022. "Equilibrium allocation strategy of multiple ESSs considering the economics and restoration capability in DNs," Applied Energy, Elsevier, vol. 306(PA).
    3. Anis ur Rehman & Muhammad Ali & Sheeraz Iqbal & Aqib Shafiq & Nasim Ullah & Sattam Al Otaibi, 2022. "Artificial Intelligence-Based Control and Coordination of Multiple PV Inverters for Reactive Power/Voltage Control of Power Distribution Networks," Energies, MDPI, vol. 15(17), pages 1-13, August.
    4. Yu, Min Gyung & Pavlak, Gregory S., 2023. "Risk-aware sizing and transactive control of building portfolios with thermal energy storage," Applied Energy, Elsevier, vol. 332(C).
    5. Ahmadi, Somayeh & Saboohi, Yadollah & Vakili, Ali, 2021. "Frameworks, quantitative indicators, characters, and modeling approaches to analysis of energy system resilience: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 144(C).
    6. Younesi, Abdollah & Shayeghi, Hossein & Wang, Zongjie & Siano, Pierluigi & Mehrizi-Sani, Ali & Safari, Amin, 2022. "Trends in modern power systems resilience: State-of-the-art review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    7. Wu, Raphael & Sansavini, Giovanni, 2020. "Integrating reliability and resilience to support the transition from passive distribution grids to islanding microgrids," Applied Energy, Elsevier, vol. 272(C).
    8. Sperstad, Iver Bakken & Kjølle, Gerd H. & Gjerde, Oddbjørn, 2020. "A comprehensive framework for vulnerability analysis of extraordinary events in power systems," Reliability Engineering and System Safety, Elsevier, vol. 196(C).
    9. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    10. Emenike, Scholastica N. & Falcone, Gioia, 2020. "A review on energy supply chain resilience through optimization," Renewable and Sustainable Energy Reviews, Elsevier, vol. 134(C).
    11. Rocchetta, Roberto, 2022. "Enhancing the resilience of critical infrastructures: Statistical analysis of power grid spectral clustering and post-contingency vulnerability metrics," Renewable and Sustainable Energy Reviews, Elsevier, vol. 159(C).
    12. Ziyi Wang & Zengqiao Chen & Cuiping Ma & Ronald Wennersten & Qie Sun, 2022. "Nationwide Evaluation of Urban Energy System Resilience in China Using a Comprehensive Index Method," Sustainability, MDPI, vol. 14(4), pages 1-36, February.
    13. Perera, A.T.D. & Hong, Tianzhen, 2023. "Vulnerability and resilience of urban energy ecosystems to extreme climate events: A systematic review and perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    14. Amrutha Raju Battula & Sandeep Vuddanti & Surender Reddy Salkuti, 2021. "Review of Energy Management System Approaches in Microgrids," Energies, MDPI, vol. 14(17), pages 1-32, September.
    15. Khawaja Haider Ali & Mohammad Abusara & Asif Ali Tahir & Saptarshi Das, 2023. "Dual-Layer Q-Learning Strategy for Energy Management of Battery Storage in Grid-Connected Microgrids," Energies, MDPI, vol. 16(3), pages 1-17, January.
    16. Sun, Qirun & Wu, Zhi & Ma, Zhoujun & Gu, Wei & Zhang, Xiao-Ping & Lu, Yuping & Liu, Pengxiang, 2022. "Resilience enhancement strategy for multi-energy systems considering multi-stage recovery process and multi-energy coordination," Energy, Elsevier, vol. 241(C).
    17. Lilia Tightiz & Joon Yoo, 2022. "A Review on a Data-Driven Microgrid Management System Integrating an Active Distribution Network: Challenges, Issues, and New Trends," Energies, MDPI, vol. 15(22), pages 1-24, November.
    18. Gonçalves, Ana & Marques, Margarida Correia & Loureiro, Sílvia & Nieto, Raquel & Liberato, Margarida L.R., 2023. "Disruption risk analysis of the overhead power lines in Portugal," Energy, Elsevier, vol. 263(PA).
    19. Zhang, Xi & Tu, Haicheng & Guo, Jianbo & Ma, Shicong & Li, Zhen & Xia, Yongxiang & Tse, Chi Kong, 2021. "Braess paradox and double-loop optimization method to enhance power grid resilience," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    20. Qiu, Dawei & Wang, Yi & Zhang, Tingqi & Sun, Mingyang & Strbac, Goran, 2023. "Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience," Applied Energy, Elsevier, vol. 336(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:15:y:2022:i:19:p:7381-:d:936094. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.