IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v17y2024i16p3990-d1454689.html
   My bibliography  Save this article

Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach

Author

Listed:
  • Xiaohan Fang

    (Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Anhui University, Hefei 230601, China
    School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Peng Hong

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Shuping He

    (Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Anhui University, Hefei 230601, China
    School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Yuhao Zhang

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Di Tan

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

Abstract

An efficient energy management system (EMS) enhances microgrid performance in terms of stability, safety, and economy. Traditional centralized or decentralized energy management systems are unable to meet the increasing demands for autonomous decision-making, privacy protection, global optimization, and rapid collaboration simultaneously. This paper proposes a hierarchical multi-layer EMS for microgrid, comprising supply layer, demand layer, and neutral scheduling layer. Additionally, common mathematical optimization methods struggle with microgrid scheduling decision problem due to challenges in mechanism modeling, supply–demand uncertainty, and high real-time and autonomy requirements. Therefore, an improved proximal policy optimization (PPO) approach is proposed for the multi-layer EMS. Specifically, in the centrally managed supply layer, a centralized PPO algorithm is utilized to determine the optimal power generation strategy. In the decentralized demand layer, an auction market is established, and multi-agent proximal policy optimization (MAPPO) algorithm with an action-guidance-based mechanism is employed for each consumer, to implement individual auction strategy. The neutral scheduling layer interacts with other layers, manages information, and protects participant privacy. Numerical results validate the effectiveness of the proposed multi-layer EMS framework and the PPO-based optimization methods.

Suggested Citation

  • Xiaohan Fang & Peng Hong & Shuping He & Yuhao Zhang & Di Tan, 2024. "Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach," Energies, MDPI, vol. 17(16), pages 1-22, August.
    • Handle: RePEc:gam:jeners:v:17:y:2024:i:16:p:3990-:d:1454689
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/17/16/3990/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/17/16/3990/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Petrollese, Mario & Valverde, Luis & Cocco, Daniele & Cau, Giorgio & Guerra, José, 2016. "Real-time integration of optimal generation scheduling with MPC for the energy management of a renewable hydrogen-based microgrid," Applied Energy, Elsevier, vol. 166(C), pages 96-106.
    2. Comodi, Gabriele & Giantomassi, Andrea & Severini, Marco & Squartini, Stefano & Ferracuti, Francesco & Fonti, Alessandro & Nardi Cesarini, Davide & Morodo, Matteo & Polonara, Fabio, 2015. "Multi-apartment residential microgrid with electrical and thermal storage devices: Experimental analysis and simulation of energy management strategies," Applied Energy, Elsevier, vol. 137(C), pages 854-866.
    3. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    4. Liu, Guodong & Jiang, Tao & Ollis, Thomas B. & Zhang, Xiaohu & Tomsovic, Kevin, 2019. "Distributed energy management for community microgrids considering network operational constraints and building thermal dynamics," Applied Energy, Elsevier, vol. 239(C), pages 83-95.
    5. Arroyo, Javier & Manna, Carlo & Spiessens, Fred & Helsen, Lieve, 2022. "Reinforced model predictive control (RL-MPC) for building energy management," Applied Energy, Elsevier, vol. 309(C).
    6. Tabar, Vahid Sohrabi & Jirdehi, Mehdi Ahmadi & Hemmati, Reza, 2017. "Energy management in microgrid based on the multi objective stochastic programming incorporating portable renewable energy resource as demand response option," Energy, Elsevier, vol. 118(C), pages 827-839.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fontenot, Hannah & Dong, Bing, 2019. "Modeling and control of building-integrated microgrids for optimal energy management – A review," Applied Energy, Elsevier, vol. 254(C).
    2. de la Hoz, Jordi & Martín, Helena & Alonso, Alex & Carolina Luna, Adriana & Matas, José & Vasquez, Juan C. & Guerrero, Josep M., 2019. "Regulatory-framework-embedded energy management system for microgrids: The case study of the Spanish self-consumption scheme," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    3. Zhang, Jingrui & Wu, Yihong & Guo, Yiran & Wang, Bo & Wang, Hengyue & Liu, Houde, 2016. "A hybrid harmony search algorithm with differential evolution for day-ahead scheduling problem of a microgrid with consideration of power flow constraints," Applied Energy, Elsevier, vol. 183(C), pages 791-804.
    4. Zia, Muhammad Fahad & Elbouchikhi, Elhoussin & Benbouzid, Mohamed, 2018. "Microgrids energy management systems: A critical review on methods, solutions, and prospects," Applied Energy, Elsevier, vol. 222(C), pages 1033-1055.
    5. Tabar, Vahid Sohrabi & Ghassemzadeh, Saeid & Tohidi, Sajjad, 2019. "Energy management in hybrid microgrid with considering multiple power market and real time demand response," Energy, Elsevier, vol. 174(C), pages 10-23.
    6. Álex Omar Topa Gavilema & José Domingo Álvarez & José Luis Torres Moreno & Manuel Pérez García, 2021. "Towards Optimal Management in Microgrids: An Overview," Energies, MDPI, vol. 14(16), pages 1-25, August.
    7. Àlex Alonso-Travesset & Helena Martín & Sergio Coronas & Jordi de la Hoz, 2022. "Optimization Models under Uncertainty in Distributed Generation Systems: A Review," Energies, MDPI, vol. 15(5), pages 1-40, March.
    8. Sharma, Pavitra & Dutt Mathur, Hitesh & Mishra, Puneet & Bansal, Ramesh C., 2022. "A critical and comparative review of energy management strategies for microgrids," Applied Energy, Elsevier, vol. 327(C).
    9. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    10. Félix González & Paul Arévalo & Luis Ramirez, 2025. "Game Theory and Robust Predictive Control for Peer-to-Peer Energy Management: A Pathway to a Low-Carbon Economy," Sustainability, MDPI, vol. 17(5), pages 1-23, February.
    11. Osorio, J.D. & Rivera-Alvarez, A. & Swain, M. & Ordonez, J.C., 2015. "Exergy analysis of discharging multi-tank thermal energy storage systems with constant heat extraction," Applied Energy, Elsevier, vol. 154(C), pages 333-343.
    12. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    13. Jingjing Zhai & Xiaobei Wu & Zihao Li & Shaojie Zhu & Bo Yang & Haoming Liu, 2021. "Day-Ahead and Intra-Day Collaborative Optimized Operation among Multiple Energy Stations," Energies, MDPI, vol. 14(4), pages 1-33, February.
    14. Coelho, Vitor N. & Coelho, Igor M. & Coelho, Bruno N. & Cohen, Miri Weiss & Reis, Agnaldo J.R. & Silva, Sidelmo M. & Souza, Marcone J.F. & Fleming, Peter J. & Guimarães, Frederico G., 2016. "Multi-objective energy storage power dispatching using plug-in vehicles in a smart-microgrid," Renewable Energy, Elsevier, vol. 89(C), pages 730-742.
    15. Dario Garozzo & Giuseppe Marco Tina, 2020. "Evaluation of the Effective Active Power Reserve for Fast Frequency Response of PV with BESS Inverters Considering Reactive Power Control," Energies, MDPI, vol. 13(13), pages 1-16, July.
    16. Zeyue Sun & Mohsen Eskandari & Chaoran Zheng & Ming Li, 2022. "Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning," Energies, MDPI, vol. 16(1), pages 1-20, December.
    17. Guodong Liu & Maximiliano F. Ferrari & Thomas B. Ollis & Kevin Tomsovic, 2022. "An MILP-Based Distributed Energy Management for Coordination of Networked Microgrids," Energies, MDPI, vol. 15(19), pages 1-20, September.
    18. Zheng, Ling & Zhou, Bin & Cao, Yijia & Wing Or, Siu & Li, Yong & Wing Chan, Ka, 2022. "Hierarchical distributed multi-energy demand response for coordinated operation of building clusters," Applied Energy, Elsevier, vol. 308(C).
    19. Ahmad Khan, Aftab & Naeem, Muhammad & Iqbal, Muhammad & Qaisar, Saad & Anpalagan, Alagan, 2016. "A compendium of optimization objectives, constraints, tools and algorithms for energy management in microgrids," Renewable and Sustainable Energy Reviews, Elsevier, vol. 58(C), pages 1664-1683.
    20. Gabriella Ferruzzi & Giorgio Graditi & Federico Rossi, 2020. "A joint approach for strategic bidding of a microgrid in energy and spinning reserve markets," Energy & Environment, , vol. 31(1), pages 88-115, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:17:y:2024:i:16:p:3990-:d:1454689. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.