Adaptive EV charging scheduling in real-time imbalance markets using model-free reinforcement learning

The growing share of intermittent renewable energy sources introduces stochastic fluctuations in supply that increase the risk of network imbalances, requiring a substantial reliance on flexible end users to maintain real-time system stability. Securing flexible resources allows Balance Responsible Parties (BRPs) to mitigate financial exposure from unintended imbalances in their energy portfolio, while granting them the discretion to strategically deviate from their scheduled positions to provide system support and receive remuneration from the system operator. The developed framework helps electric vehicles (EVs) to participate directly in the imbalance market while accounting for uncertainties in imbalance price forecasting and departure times. A model-free reinforcement learning approach based on Proximal Policy Optimization (PPO) is employed to support decision-making under these uncertain conditions. This study evaluates its performance using 2023 Belgian imbalance price data in an office parking facility. Results show that the PPO agent handles the uncertainty effectively and shifts EV charging from a cost-driven operation to a revenue-generating one. Crucially, the study finds that policies trained without departure time uncertainty outperform those trained under stochastic departure, suggesting that an aggressive arbitrage strategy outweighs the benefits of conservative uncertainty modeling. Furthermore, adapted policy in imperfect 1-minute price signals reduced flexibility service volume by 5%, but impacted financial returns by less than 1%, meaning imbalance price noise primarily limits the reliable volume of flexibility services rather than economic efficiency. Results from the policy analysis show that the agent maximizes returns by adopting binary control logic, toggling charging on-off rather than modulating power levels.