Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Coordination of Distributed Energy Resources through Local Energy Markets and Deep Reinforcement Learning

Daniel May, Matthew Taylor, Petr Musilek

TL;DR

This study demonstrates the effectiveness of DRL in decentralized grid management, highlighting its scalability and near-optimal performance in reducing net load variability within community-driven energy markets.

Abstract

As distributed energy resources (DERs) grow, the electricity grid faces increased net load variability at the grid edge, impacting operability and reliability. Transactive energy, facilitated through local energy markets, offers a decentralized, indirect demand response solution, with model-free control techniques, such as deep reinforcement learning (DRL), enabling automated, decentralized participation. However, existing studies largely overlook community-level net load variability, focusing instead on socioeconomic metrics. This study addresses this gap by using DRL agents to automate end-user participation in a local energy market (ALEX), where agents act independently to minimize individual energy bills. Results reveal a strong link between bill reduction and decreased net load variability, assessed across metrics such as ramping rate, load factor, and peak demand over various time horizons. Using a no-control baseline, DRL agents are benchmarked against a near-optimal dynamic programming approach. The dynamic programming benchmark achieves reductions of 22.05 percent, 83.92 percent, and 24.09 percent in daily import, export, and peak demand, respectively, while the DRL agents show comparable or superior results with reductions of 21.93 percent, 84.46 percent, and 27.02 percent. This study demonstrates the effectiveness of DRL in decentralized grid management, highlighting its scalability and near-optimal performance in reducing net load variability within community-driven energy markets.

Decentralized Coordination of Distributed Energy Resources through Local Energy Markets and Deep Reinforcement Learning

TL;DR

This study demonstrates the effectiveness of DRL in decentralized grid management, highlighting its scalability and near-optimal performance in reducing net load variability within community-driven energy markets.

Abstract

As distributed energy resources (DERs) grow, the electricity grid faces increased net load variability at the grid edge, impacting operability and reliability. Transactive energy, facilitated through local energy markets, offers a decentralized, indirect demand response solution, with model-free control techniques, such as deep reinforcement learning (DRL), enabling automated, decentralized participation. However, existing studies largely overlook community-level net load variability, focusing instead on socioeconomic metrics. This study addresses this gap by using DRL agents to automate end-user participation in a local energy market (ALEX), where agents act independently to minimize individual energy bills. Results reveal a strong link between bill reduction and decreased net load variability, assessed across metrics such as ramping rate, load factor, and peak demand over various time horizons. Using a no-control baseline, DRL agents are benchmarked against a near-optimal dynamic programming approach. The dynamic programming benchmark achieves reductions of 22.05 percent, 83.92 percent, and 24.09 percent in daily import, export, and peak demand, respectively, while the DRL agents show comparable or superior results with reductions of 21.93 percent, 84.46 percent, and 27.02 percent. This study demonstrates the effectiveness of DRL in decentralized grid management, highlighting its scalability and near-optimal performance in reducing net load variability within community-driven energy markets.
Paper Structure (12 sections, 10 equations, 3 figures, 2 tables)

This paper contains 12 sections, 10 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. Related Literature
  4. Autonomous Local Energy eXchange
  5. Reinforcement Learning and Markov Decision Processes
  6. Methodology and Evaluation
  7. Autonomous Local Energy eXchange as Markov Decision Process
  8. Shared Experience Recurrent Proximal Policy Optimization
  9. Experimental Design
  10. Results and Discussion
  11. Conclusion
  12. Auxiliary Information regarding Algorithm Setup

Figures (3)

  • Figure 1: Agent to environment interaction diagram, taken from Sutton & Barto Sutton.
  • Figure 2: Average participant bill savings comparison between ALEX RL (blue), ALEX DP (red). Shaded areas depict variance bands between a set of 5 ALEX RL runs, trained over 117 episodes.
  • Figure 3: Performance of recorded community-level metrics per episode throughout training. The opaque scattered data points represent singular episode equivalents, while the blue line depicts the metric performance of the most recent highest return achieved.