Distributed Reinforcement Learning using Local Smart Meter Data for Voltage Regulation in Distribution Networks

TL;DR

The paper tackles voltage regulation in distribution networks under privacy and data-sharing constraints by proposing a distributed reinforcement learning framework that uses local smart-meter data. Each household ESS operates as an independent agent with a local Thevenin-based environment, supported by a piecewise function and a Transformer-based sensitivity model to correct voltage estimates and action effects. An online coordination layer with a coordination scaler and a local optimization step mitigates cross-agent interference and prevents excessive simultaneous actions. Case studies on a CIGRE LV network show that the distributed approach achieves effective voltage regulation with faster training and preserved privacy, while coordination reduces system-wide violations and improves reliability. Overall, the work advances scalable, privacy-preserving voltage control in distribution networks through distributed RL, physics-informed estimation, and lightweight coordination.

Abstract

Centralised reinforcement learning (RL) for voltage magnitude regulation in distribution networks typically involves numerous agent-environment interactions and power flow (PF) calculations, inducing computational overhead and privacy concerns over shared data. Thus, we propose a distributed RL algorithm to regulate voltage magnitude. First, a dynamic Thevenin equivalent model is integrated within smart meters (SM), enabling local voltage magnitude estimation using local SM data for RL agent training, and mitigating the dependency of synchronised data collection and centralised PF calculations. To mitigate estimation errors induced by Thevenin model inaccuracies, a voltage magnitude correction strategy that combines piecewise functions and neural networks is introduced. The piecewise function corrects the large errors of estimated voltage magnitude, while a neural network mimics the grid's sensitivity to control actions, improving action adjustment precision. Second, a coordination strategy is proposed to refine local RL agent actions online, preventing voltage magnitude violations induced by excessive actions from multiple independently trained agents. Case studies on energy storage systems validate the feasibility and effectiveness of the proposed approach, demonstrating its potential to improve voltage regulation in distribution networks.

Figures (9)

• Figure 1: Illustrative example of distributed environment construction for smart agents in the LVDN with two ESSs.
• Figure 2: Coordinated-Distributed Framework for ESS-based Voltage pre-active management
• Figure 3: Framework of the proposed distributed RL algorithm: (a) local voltage estimation, and (b) RL agent training in the constructed local environment.
• Figure 4: Topology of the 18-bus CIGRE LVND.
• Figure 5: Local voltage magnitude estimation and errors distribution: (a)-(c) estimated voltage magnitude, (d) loss of the ANNs training, and (e) distribution of relative errors.