Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

Shuyi Gao, Stavros Orfanoudakis, Shengren Hou, Peter Palensky, Pedro P. Vergara

Abstract

Optimal dispatch of energy storage systems (ESSs) in distribution networks involves jointly improving operating economy and voltage security under time-varying conditions and possible topology changes. To support fast online decision making, we develop a topology-aware Reinforcement Learning architecture based on Twin Delayed Deep Deterministic Policy Gradient (TD3), which integrates graph neural networks (GNNs) as graph feature encoders for ESS dispatch. We conduct a systematic investigation of three GNN variants: graph convolutional networks (GCNs), topology adaptive graph convolutional networks (TAGConv), and graph attention networks (GATs) on the 34-bus and 69-bus systems, and evaluate robustness under multiple topology reconfiguration cases as well as cross-system transfer between networks with different system sizes. Results show that GNN-based controllers consistently reduce the number and magnitude of voltage violations, with clearer benefits on the 69-bus system and under reconfiguration; on the 69-bus system, TD3-GCN and TD3-TAGConv also achieve lower saved cost relative to the NLP benchmark than the NN baseline. We also highlight that transfer gains are case-dependent, and zero-shot transfer between fundamentally different systems results in notable performance degradation and increased voltage magnitude violations. This work is available at: https://github.com/ShuyiGao/GNNs_RL_ESSs and https://github.com/distributionnetworksTUDelft/GNNs_RL_ESSs.

Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

Abstract

Optimal dispatch of energy storage systems (ESSs) in distribution networks involves jointly improving operating economy and voltage security under time-varying conditions and possible topology changes. To support fast online decision making, we develop a topology-aware Reinforcement Learning architecture based on Twin Delayed Deep Deterministic Policy Gradient (TD3), which integrates graph neural networks (GNNs) as graph feature encoders for ESS dispatch. We conduct a systematic investigation of three GNN variants: graph convolutional networks (GCNs), topology adaptive graph convolutional networks (TAGConv), and graph attention networks (GATs) on the 34-bus and 69-bus systems, and evaluate robustness under multiple topology reconfiguration cases as well as cross-system transfer between networks with different system sizes. Results show that GNN-based controllers consistently reduce the number and magnitude of voltage violations, with clearer benefits on the 69-bus system and under reconfiguration; on the 69-bus system, TD3-GCN and TD3-TAGConv also achieve lower saved cost relative to the NLP benchmark than the NN baseline. We also highlight that transfer gains are case-dependent, and zero-shot transfer between fundamentally different systems results in notable performance degradation and increased voltage magnitude violations. This work is available at: https://github.com/ShuyiGao/GNNs_RL_ESSs and https://github.com/distributionnetworksTUDelft/GNNs_RL_ESSs.

Paper Structure

This paper contains 17 sections, 31 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Mathematical Formulation and MDP for Optimal ESSs Dispatch
  3. Mathematical formulation
  4. ESSs optimal dispatch as a Markov Decision Process
  5. GNN-Based Reinforcement Learning for Optimal ESSs Dispatch
  6. Graph Feature Encoder for Distribution Network State
  7. Graph Massage Passing Layer
  8. Graph Convolutional Network
  9. Topology Adaptive Graph Convolution
  10. Graph Attention Network
  11. Reinforcement Learning Algorithm
  12. Case Study
  13. Performance of RLs on Training Topology
  14. Generalization to Re-configured Topologies
  15. In-network reconfiguration
  16. ...and 2 more sections

Figures (10)

  • Figure 1: Architecture of the proposed GNN-based RL framework for distribution-network ESS optimal dispatch, consisting of the graph feature encoder, the ESS dispatch environment, the experience replay buffer, and the actor–critic networks.
  • Figure 2: The average reward of RL algorithms during 1000 episodes of training on (a)34 buses system, (b)69 buses system.
  • Figure 3: Comparison of RL algorithms on 34 buses system, (a)electricity price and ESS nodes' active power, (b)-(f) ESS nodes' SOC and charging/discharging strategy of NLP, TD3-NN, TD3-GCN, TD3-TAGConv, TD3-GATConv on Node 34.
  • Figure 4: ESS nodes' voltage magnitude on 34 buses system, (a)without ESS control, (b)NLP, (c)TD3-NN, (d)TD3-GCN, (e)TD3-TAGConv, (f)TD3-GATConv.
  • Figure 5: 34 buses system RL algorithms' operational accuracy [$\%$] on topological variations
  • ...and 5 more figures