Parametric study of adaptive reinforcement learning for battery operations in microgrids

Reinforcement learning (RL) has been increasingly used for efficient energy management systems (EMSs) in microgrids. The battery storage system in the microgrid can be controlled using efficient policies derived from RL. However, little attention has been paid so far to the parametric study, which is a fundamental step for efficient implementation of such RL algorithms. Unlike previous works which focused on the implementation of different RL algorithms, this paper mainly demonstrates the parametric sensitivity study of the RL algorithms. It involves investigating the effects of (1) controllable state discretization, (2) exogenous state discretization, (3) action discretization, (4) exploration and exploitation parameters, and (5) decision intervals. Moreover, the performance of the ε-greedy randomized RL algorithm is compared against the adaptive Q-learning, derived from the adaptive approximate dynamic programming (ADP). In many microgrids utilizing solar energy and battery storage, energy management still relies on manually tuned and inefficient algorithms. This is largely due to the sensitivity of RL algorithm parameters to factors such as the specific EMS problem, environment, action-state discretization, exploration parameter and time step. We show the univariate and multivariate kernel density estimate (KDE) plots to study the RL algorithms’ performance concerning the rewards and variation of the battery state of charge (SoC) and the net power imported from the grid. Overall, the deterministic adaptive RL performs better as compared to the randomized ε-greedy algorithms in terms of rewards and simulation time. Higher discretization levels in the action space affect the convergence rate while lower discretization levels in the state space influence the performance of the algorithm. The proposed parametric analysis can be easily adapted to other EMS in more complex microgrids.