Optimizing energy management of smart grid using reinforcement learning aided by surrogate models built using physics-informed neural networks

Optimizing the energy management within a smart grid scenario presents significant challenges, primarily due to the complexity of real-world systems and the intricate interactions among various components. Reinforcement Learning (RL) is gaining prominence as a solution for addressing the challenges of Optimal Power Flow (OPF) in smart grids. However, RL needs to iterate compulsively throughout a given environment to obtain the optimal policy. This means obtaining samples from a, most likely, costly simulator, which can lead to a sample efficiency problem. In this work, we address this problem by substituting costly smart grid simulators with surrogate models built using Physics-Informed Neural Networks (PINNs), optimizing the RL policy training process by arriving at convergent results in a fraction of the time employed by the original environment. Specifically, we tested the performance of our PINN surrogate against other state-of-the-art data-driven surrogates and found that the understanding of the underlying physical nature of the problem makes the PINN surrogate the only method we studied capable of learning a good RL policy, in addition to not having to use samples from the real simulator. Our work shows that, by employing PINN surrogates, we can improve training speed by 50 %, compared to training the RL policy without using any surrogate model, enabling us to achieve results with scores on par with the original simulator more rapidly.