Physics-Informed Reinforcement Learning for Large-Scale EV Smart Charging Considering Distribution Network Voltage Constraints

TL;DR

This work tackles voltage stability in distribution networks under large-scale EV charging. It introduces PI-TD3, a physics-informed reinforcement learning algorithm that embeds differentiable power-flow calculations and EV battery SoC dynamics into the TD3 framework, enabling real-time, grid-aware charging decisions. The approach yields faster convergence, improved sample efficiency, and robust voltage regulation, closely approaching an oracle MPC on IEEE 34-bus and 123-bus networks while scaling to hundreds of EVs. The results demonstrate strong generalization to unseen load profiles and reliable performance with low variability, indicating practical potential for city-scale EV charging coordination under voltage constraints.

Abstract

Electric Vehicles (EVs) offer substantial flexibility for grid services, yet large-scale, uncoordinated charging can threaten voltage stability in distribution networks. Existing Reinforcement Learning (RL) approaches for smart charging often disregard physical grid constraints or have limited performance for complex large-scale tasks, limiting their scalability and real-world applicability. This paper introduces a physics-informed (PI) RL algorithm that integrates a differentiable power flow model and voltage-based reward design into the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, enabling EVs to deliver real-time voltage support while meeting user demands. The resulting PI-TD3 algorithm achieves faster convergence, improved sample efficiency, and reliable voltage magnitude regulation under uncertain and overloaded conditions. Benchmarks on the IEEE 34-bus and 123-bus networks show that the proposed PI-TD3 outperforms both model-free RL and optimization-based baselines in grid constraint management, user satisfaction, and economic metrics, even as the system scales to hundreds of EVs. These advances enable robust, scalable, and practical EV charging strategies that enhance grid resilience and support distribution networks operation.

Figures (7)

• Figure 1: Overview of the proposed problem setting illustrating an example distribution network with V2G charging stations, PV generation, and dynamic demand. The DSO provides real-time grid topology, demand, and PV generation data, while the CPO coordinates the charging of hundreds of EVs based on energy prices, respecting user constraints and enabling distribution network voltage support.
• Figure 2: The policy network $\pi_\theta$ generates actions that, together with the known environment dynamics (e.g., SoC and voltage updates) and sampled exogenous variables (e.g., loads, prices, and EV arrivals), are used to simulate $K$-step rollouts through the differentiable transition $T(s,a)$ and reward $R(s,a)$ functions. This enables direct gradient propagation from cumulative rewards back through the rollout.
• Figure 3: Evaluation reward performance comparison during training averaged over five random seeds.
• Figure 4: Impact of EV charging algorithm on the bus voltage magnitude and in EV charging. Bus 22 is demonstrated here as a node where applying the proposed algorithm effectively mitigates voltage limit violations.
• Figure 5: Distribution of bus voltage magnitudes for each charging algorithm. Box plots show median, interquartile range, and full voltage range over an experimental evaluation scenario; the red dashed line indicates the operational limit ($0.95$ p.u.).