Toward resilient distribution system via microgrid formation: A safe hierarchical hybrid reinforcement learning approach

During utility power outages, the distribution system can be reconfigured into microgrids (MGs) to support load restoration and enhance system resilience. Microgrid formation (MGF) involves discrete topology reconfiguration via remote-controlled switches (RCSs), and continuous power dispatch via distributed energy resources (DERs), resulting in a discrete-continuous hybrid action space. However, high-dimensional hybrid action spaces and strict physical constraints bring great challenges to learn high-performance control policies for MGF. To address it, this paper develops a two-level safe hierarchical hybrid reinforcement learning (SHHRL) approach to learn MGF policy directly and effectively within the hybrid action space. The hierarchical architecture decomposes the complex MGF problem into two subproblems: the topology reconfiguration problem and the power dispatch problem. This decomposition enables each agent to learn within a reduced subspace, thereby reducing learning complexity and improving convergence performance. In addition, expert knowledge is leveraged to ensure strict compliance with physical constraints. Specifically, an invalid action masking layer is designed to filter out infeasible discrete actions, and a safety projection layer is introduced to correct unsafe continuous actions. Furthermore, to mitigate instability caused by inter-agent dependencies and enhance learning efficiency, multi-prioritized experience replay (MPER) is incorporated. Experimental results demonstrate that the proposed approach outperforms state-of-the-art hybrid action space RL methods and effectively guarantees operational safety during restoration.