IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v335y2025ics0360544225035340.html

Enhancing self-consumption ratio in a smart microgrid by applying a reinforcement learning-based energy management system

Author

Listed:
  • Hajialigol, Parisa
  • Nweye, Kingsley
  • Aghaei, Mohammadreza
  • Najafi, Behzad
  • Moazami, Amin
  • Nagy, Zoltan

Abstract

This study presents an updated version of the CityLearn Gym environment by integrating a stochastic data-driven vehicle-to-building model. To this end, EVs are modeled as local mobile storage using stochastic behavior derived from a real-world charging dataset, considering uncertainties in EV arrival/departure times, battery capacity, and the arrival state of charge (SoC). Then, the model is integrated within CityLearn to use a reinforcement learning-based energy management system (EMS) to control and optimize a smart microgrid's energy consumption and storage systems. A real-world microgrid in Norway is used to evaluate system performance under three scenarios, including one where solar panel (PV) generation is shared across buildings. The main objective is to provide energy flexibility by enhancing the self-energy consumption of solar generation by finding the optimal control policy for storage systems, which are batteries and EVs. The proposed EMS is designed using the soft actor-critic (SAC) algorithm to coordinate among the different flexible sources by defining the priority resources and direct charging control signals. Three scenarios are investigated and the shared scenario, which in PV generation can be shared between buildings, has had the best performance. The performance of the EMS is evaluated by five key indicators. The results show that the self-consumption ratio of microgrid has been increased up to 23 % and daily peak power has been reduced by up to 20 % compared to RBC as a conventional method. This highlights the impact of storage systems, especially EVs, on the microgrid performance to increase the penetration of solar energy through the energy transition and the potential of RL in advancing intelligent EMS design for future energy systems.

Suggested Citation

  • Hajialigol, Parisa & Nweye, Kingsley & Aghaei, Mohammadreza & Najafi, Behzad & Moazami, Amin & Nagy, Zoltan, 2025. "Enhancing self-consumption ratio in a smart microgrid by applying a reinforcement learning-based energy management system," Energy, Elsevier, vol. 335(C).
    • Handle: RePEc:eee:energy:v:335:y:2025:i:c:s0360544225035340
    DOI: 10.1016/j.energy.2025.137892
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544225035340
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2025.137892?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Polimeni, Simone & Moretti, Luca & Martelli, Emanuele & Leva, Sonia & Manzolini, Giampaolo, 2023. "A novel stochastic model for flexible unit commitment of off-grid microgrids," Applied Energy, Elsevier, vol. 331(C).
    2. Zhang, Bin & Hu, Weihao & Xu, Xiao & Li, Tao & Zhang, Zhenyuan & Chen, Zhe, 2022. "Physical-model-free intelligent energy management for a grid-connected hybrid wind-microturbine-PV-EV energy system via deep reinforcement learning approach," Renewable Energy, Elsevier, vol. 200(C), pages 433-448.
    3. Han, Lijin & Yang, Ke & Ma, Tian & Yang, Ningkang & Liu, Hui & Guo, Lingxiong, 2022. "Battery life constrained real-time energy management strategy for hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 259(C).
    4. Sunyong Kim & Hyuk Lim, 2018. "Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings," Energies, MDPI, vol. 11(8), pages 1-19, August.
    5. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    6. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    7. Kaur, Amandeep & Kaushal, Jitender & Basak, Prasenjit, 2016. "A review on microgrid central controller," Renewable and Sustainable Energy Reviews, Elsevier, vol. 55(C), pages 338-345.
    8. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    9. Hirschburger, Rafael & Weidlich, Anke, 2020. "Profitability of photovoltaic and battery systems on municipal buildings," Renewable Energy, Elsevier, vol. 153(C), pages 1163-1173.
    10. Ottesen, Stig Odegaard & Tomasgard, Asgeir, 2015. "A stochastic model for scheduling energy flexibility in buildings," Energy, Elsevier, vol. 88(C), pages 364-376.
    11. Park, Keonwoo & Moon, Ilkyeong, 2022. "Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid," Applied Energy, Elsevier, vol. 328(C).
    12. Nallapaneni Manoj Kumar & Aneesh A. Chand & Maria Malvoni & Kushal A. Prasad & Kabir A. Mamun & F.R. Islam & Shauhrat S. Chopra, 2020. "Distributed Energy Resources and the Application of AI, IoT, and Blockchain in Smart Grids," Energies, MDPI, vol. 13(21), pages 1-42, November.
    13. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    14. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    15. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Panagiotis Michailidis & Iakovos Michailidis & Elias Kosmatopoulos, 2025. "Reinforcement Learning for Optimizing Renewable Energy Utilization in Buildings: A Review on Applications and Innovations," Energies, MDPI, vol. 18(7), pages 1-40, March.
    3. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    4. Soleimanzade, Mohammad Amin & Kumar, Amit & Sadrzadeh, Mohtada, 2022. "Novel data-driven energy management of a hybrid photovoltaic-reverse osmosis desalination system using deep reinforcement learning," Applied Energy, Elsevier, vol. 317(C).
    5. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    6. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    7. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    8. Gerald Jones & Xueping Li & Yulin Sun, 2024. "Robust Energy Management Policies for Solar Microgrids via Reinforcement Learning," Energies, MDPI, vol. 17(12), pages 1-22, June.
    9. Bio Gassi, Karim & Baysal, Mustafa, 2023. "Improving real-time energy decision-making model with an actor-critic agent in modern microgrids with energy storage devices," Energy, Elsevier, vol. 263(PE).
    10. Mahmud, Sakib & Sayed, Aya Nabil & Himeur, Yassine & Nhlabatsi, Armstrong & Bensaali, Faycal, 2026. "A comprehensive review of deep reinforcement learning applications from centralized power generation to modern energy internet frameworks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PE).
    11. Zhou, Yuekuan & Liu, Xiaohua & Zhao, Qianchuan, 2024. "A stochastic vehicle schedule model for demand response and grid flexibility in a renewable-building-e-transportation-microgrid," Renewable Energy, Elsevier, vol. 221(C).
    12. Li, Yang & Zhao, Bingsong & Li, Yuanzheng & Long, Chao & Li, Sen & Dong, Zhaoyang & Shahidehpour, Mohammad, 2025. "Safe-AutoSAC: AutoML-enhanced safe deep reinforcement learning for integrated energy system scheduling with multi-channel informer forecasting and electric vehicle demand response," Applied Energy, Elsevier, vol. 399(C).
    13. Yang, Meng & Chen, Yue & Huang, Shihan & Chen, Laijun, 2025. "Recent advances in coordination and optimization of power-transportation systems: An overview," Renewable and Sustainable Energy Reviews, Elsevier, vol. 220(C).
    14. Van-Hai Bui & Akhtar Hussain & Hak-Man Kim, 2019. "Q-Learning-Based Operation Strategy for Community Battery Energy Storage System (CBESS) in Microgrid System," Energies, MDPI, vol. 12(9), pages 1-17, May.
    15. Panagiotis Michailidis & Iakovos Michailidis & Elias Kosmatopoulos, 2025. "Reinforcement Learning for Electric Vehicle Charging Management: Theory and Applications," Energies, MDPI, vol. 18(19), pages 1-50, October.
    16. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    17. Xu, Xuesong & Xu, Kai & Zeng, Ziyang & Tang, Jiale & He, Yuanxing & Shi, Guangze & Zhang, Tao, 2024. "Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 375(C).
    18. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    19. Emily van Huffelen & Roel Brouwer & Marjan van den Akker, 2025. "Grid-Constrained Online Scheduling of Flexible Electric Vehicle Charging," Energies, MDPI, vol. 18(19), pages 1-20, September.
    20. Khawaja Haider Ali & Mohammad Abusara & Asif Ali Tahir & Saptarshi Das, 2023. "Dual-Layer Q-Learning Strategy for Energy Management of Battery Storage in Grid-Connected Microgrids," Energies, MDPI, vol. 16(3), pages 1-17, January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:335:y:2025:i:c:s0360544225035340. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.