A novel two-level deep reinforcement learning enabled game approach for incentive-based distributed voltage regulation with participation of autonomous photovoltaic inverters

The voltage violation issues caused by uncertainties of renewable energy generation severely restrict the operation of active distribution networks (ADN). Meanwhile, current voltage regulation methods using PV inverters always ignore the autonomy of PV prosumers. In this context, this paper proposes a novel two-level deep reinforcement learning (DRL) enabled incentive-based voltage regulation approach to mitigate voltage fluctuations while fully exploiting the potential of PV prosumers. To conduct voltage regulation in a distributed manner, the ADN is first partitioned into several sub-networks based on voltage-reactive sensitivity matrix. In the outer-level model, the distributed system operator (DSO) aims to select suitable compensation prices for different sub-networks to incentivize within prosumers to participate in voltage regulation, which is formulated as a Markov decision process (MDP) and solved by a centralized agent based soft actor-critic (SAC) algorithm. In the inner-level model, the autonomous prosumers within each sub-network feedback their corresponding voltage regulation strategies including both reactive power and active curtailment to maximize their own profits, which is modeled as Markov games and solved by a multi-agent SAC (MASAC) algorithm. To stabilize the training process and improve the computational efficiency, the inner-level agents are pre-trained and then the two-level agents are trained concurrently to solve the game problem. Case studies are conducted based on the IEEE 33-bus system, and the numerical results demonstrate that the proposed method can achieve effective performance on voltage regulation while guaranteeing prosumers’ profit.