Two-Stage Optimization of Virtual Power Plant Operation Considering Substantial Quantity of EVs Participation Using Reinforcement Learning and Gradient-Based Programming

Modern electrical vehicles (EVs) are equipped with sizable batteries that possess significant potential as energy prosumers. EVs are poised to be transformative assets and pivotal contributors to the virtual power plant (VPP), enhancing the performance and profitability of VPPs. The number of household EVs is increasing yearly, and this poses new challenges to the optimization of VPP operations. The computational cost increases exponentially as the number of decision variables rises with the increasing participation of EVs. This paper explores the role of a large number of EVs as prosumers, interacting with a VPP consisting of a photovoltaic system and battery energy storage system. To accommodate the large quantity of EVs in the modeling, this research adopts the decentralized control structure. It optimizes EV operations by regulating their charging and discharging behavior in response to pricing signals from the VPP. A two-stage optimization framework is proposed for VPP-EV operation using a reinforcement algorithm and gradient-based programming. Action masking for reinforcement learning is explored to eliminate invalid actions, reducing ineffective exploration, thereby accelerating the convergence of the algorithm. The proposed approach is capable of handling a substantial number of EVs and addressing the stochastic characteristics of EV charging and discharging behaviors. Simulation results demonstrate that the VPP-EV operation optimization increases the revenue of the VPP and significantly reduces the electricity costs for EV owners. Through the optimization of EV operations, the charging cost of 1000 EVs participating in the V2G services is reduced by 26.38% compared to those that opt out of the scheme, and VPP revenue increases by 27.83% accordingly.