Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RL-ADN: A High-Performance Deep Reinforcement Learning Environment for Optimal Energy Storage Systems Dispatch in Active Distribution Networks

Shengren Hou, Shuyi Gao, Weijie Xia, Edgar Mauricio Salazar Duque, Peter Palensky, Pedro P. Vergara

TL;DR

RL-ADN addresses the need for a flexible, scalable, and efficient reinforcement learning environment for optimal ESS dispatch in active distribution networks. It introduces GMC-based data augmentation and a Laurent power flow solver to substantially increase training diversity and reduce computation time, respectively. The framework supports multiple SOTA DRL algorithms and provides a standardized 34-node benchmark, demonstrating improved performance and scalability. The work highlights significant practical impact by enabling near-optimal ESS dispatch with much faster training, facilitating large-scale deployment and research in distribution-network operations.

Abstract

Deep Reinforcement Learning (DRL) presents a promising avenue for optimizing Energy Storage Systems (ESSs) dispatch in distribution networks. This paper introduces RL-ADN, an innovative open-source library specifically designed for solving the optimal ESSs dispatch in active distribution networks. RL-ADN offers unparalleled flexibility in modeling distribution networks, and ESSs, accommodating a wide range of research goals. A standout feature of RL-ADN is its data augmentation module, based on Gaussian Mixture Model and Copula (GMC) functions, which elevates the performance ceiling of DRL agents. Additionally, RL-ADN incorporates the Laurent power flow solver, significantly reducing the computational burden of power flow calculations during training without sacrificing accuracy. The effectiveness of RL-ADN is demonstrated using in different sizes of distribution networks, showing marked performance improvements in the adaptability of DRL algorithms for ESS dispatch tasks. This enhancement is particularly beneficial from the increased diversity of training scenarios. Furthermore, RL-ADN achieves a tenfold increase in computational efficiency during training, making it highly suitable for large-scale network applications. The library sets a new benchmark in DRL-based ESSs dispatch in distribution networks and it is poised to advance DRL applications in distribution network operations significantly. RL-ADN is available at: https://github.com/ShengrenHou/RL-ADN and https://github.com/distributionnetworksTUDelft/RL-ADN.

RL-ADN: A High-Performance Deep Reinforcement Learning Environment for Optimal Energy Storage Systems Dispatch in Active Distribution Networks

TL;DR

RL-ADN addresses the need for a flexible, scalable, and efficient reinforcement learning environment for optimal ESS dispatch in active distribution networks. It introduces GMC-based data augmentation and a Laurent power flow solver to substantially increase training diversity and reduce computation time, respectively. The framework supports multiple SOTA DRL algorithms and provides a standardized 34-node benchmark, demonstrating improved performance and scalability. The work highlights significant practical impact by enabling near-optimal ESS dispatch with much faster training, facilitating large-scale deployment and research in distribution-network operations.

Abstract

Deep Reinforcement Learning (DRL) presents a promising avenue for optimizing Energy Storage Systems (ESSs) dispatch in distribution networks. This paper introduces RL-ADN, an innovative open-source library specifically designed for solving the optimal ESSs dispatch in active distribution networks. RL-ADN offers unparalleled flexibility in modeling distribution networks, and ESSs, accommodating a wide range of research goals. A standout feature of RL-ADN is its data augmentation module, based on Gaussian Mixture Model and Copula (GMC) functions, which elevates the performance ceiling of DRL agents. Additionally, RL-ADN incorporates the Laurent power flow solver, significantly reducing the computational burden of power flow calculations during training without sacrificing accuracy. The effectiveness of RL-ADN is demonstrated using in different sizes of distribution networks, showing marked performance improvements in the adaptability of DRL algorithms for ESS dispatch tasks. This enhancement is particularly beneficial from the increased diversity of training scenarios. Furthermore, RL-ADN achieves a tenfold increase in computational efficiency during training, making it highly suitable for large-scale network applications. The library sets a new benchmark in DRL-based ESSs dispatch in distribution networks and it is poised to advance DRL applications in distribution network operations significantly. RL-ADN is available at: https://github.com/ShengrenHou/RL-ADN and https://github.com/distributionnetworksTUDelft/RL-ADN.
Paper Structure (34 sections, 13 equations, 9 figures, 4 tables)

This paper contains 34 sections, 13 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Related Work
  4. Contributions
  5. Background
  6. Optimal ESS dispatch tasks in distribution networks
  7. MDP formulation and reinforcement learning
  8. RL-ADN Framework
  9. Overview
  10. Data Source Layer
  11. Configuration Layer
  12. Data Manager
  13. Data Augmentation
  14. Distribution Network Simulator
  15. Interaction Loop Layer
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Architecture of policy-based DRL algorithms. (a) Deep Deterministic Policy Gradient (DDPG), (b) Twin Delayed DDPG (TD3), (c) Proximal Policy Optimization (PPO), (d) Soft Actor-Critic (SAC).
  • Figure 2: Framework of the RL-ADN package. Configuration data for the distribution network and the ESSs are selected from data sources. Subsequently, corresponding time-series data undergo preprocessing. Through Configuration Layer, the environment is constituted of the distribution network, ESSs, and data manager.
  • Figure 3: Modified IEEE-34 Node bus test system with distributed PV generation and EESs. The ESSs are placed at the end of each feeder to increase the number of voltage magnitude issues experienced.
  • Figure 4: $(a)$ Average total reward as in \ref{['eq:new_penalty_reward']}. $(b)$ Operational cost or first term of reward in \ref{['eq:new_penalty_reward']}. $(c)$ Cumulative penalty for voltage magnitude violations or second term of reward in \ref{['eq:new_penalty_reward']}, all during training.
  • Figure 5: Dispatch decisions obtained by DRL algorithms and NLP for the ESS connected to node 16
  • ...and 4 more figures