Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Peer-to-Peer Energy Trading of Solar and Energy Storage: A Networked Multiagent Reinforcement Learning Approach

Chen Feng, Andrew L. Liu

TL;DR

The paper tackles the challenge of enabling scalable, decentralized peer-to-peer energy trading in distribution networks where renewables have zero marginal cost. It couples a SDR-based market-clearing mechanism with a consensus-based MARL framework to automate bidding by prosumers while enforcing physical network constraints such as voltage limits. The authors prove convergence under a set of assumptions and demonstrate via IEEE 13-bus simulations that consensus-MARL outperforms fully decentralized PPO and MADDPG in long-term rewards, voltage regulation, and line-capacity feasibility. This work offers a practical, privacy-preserving path toward real-world P2P trading by integrating market design with network-aware learning at the grid edge.

Abstract

Utilizing distributed renewable and energy storage resources in local distribution networks via peer-to-peer (P2P) energy trading has long been touted as a solution to improve energy systems' resilience and sustainability. Consumers and prosumers (those who have energy generation resources), however, do not have the expertise to engage in repeated P2P trading, and the zero-marginal costs of renewables present challenges in determining fair market prices. To address these issues, we propose multi-agent reinforcement learning (MARL) frameworks to help automate consumers' bidding and management of their solar PV and energy storage resources, under a specific P2P clearing mechanism that utilizes the so-called supply-demand ratio. In addition, we show how the MARL frameworks can integrate physical network constraints to realize voltage control, hence ensuring physical feasibility of the P2P energy trading and paving way for real-world implementations.

Peer-to-Peer Energy Trading of Solar and Energy Storage: A Networked Multiagent Reinforcement Learning Approach

TL;DR

The paper tackles the challenge of enabling scalable, decentralized peer-to-peer energy trading in distribution networks where renewables have zero marginal cost. It couples a SDR-based market-clearing mechanism with a consensus-based MARL framework to automate bidding by prosumers while enforcing physical network constraints such as voltage limits. The authors prove convergence under a set of assumptions and demonstrate via IEEE 13-bus simulations that consensus-MARL outperforms fully decentralized PPO and MADDPG in long-term rewards, voltage regulation, and line-capacity feasibility. This work offers a practical, privacy-preserving path toward real-world P2P trading by integrating market design with network-aware learning at the grid edge.

Abstract

Utilizing distributed renewable and energy storage resources in local distribution networks via peer-to-peer (P2P) energy trading has long been touted as a solution to improve energy systems' resilience and sustainability. Consumers and prosumers (those who have energy generation resources), however, do not have the expertise to engage in repeated P2P trading, and the zero-marginal costs of renewables present challenges in determining fair market prices. To address these issues, we propose multi-agent reinforcement learning (MARL) frameworks to help automate consumers' bidding and management of their solar PV and energy storage resources, under a specific P2P clearing mechanism that utilizes the so-called supply-demand ratio. In addition, we show how the MARL frameworks can integrate physical network constraints to realize voltage control, hence ensuring physical feasibility of the P2P energy trading and paving way for real-world implementations.
Paper Structure (13 sections, 2 theorems, 29 equations, 9 figures, 1 algorithm)

This paper contains 13 sections, 2 theorems, 29 equations, 9 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Market Clearing Mechanism
  4. Decentralized Learning with Consensus Updates
  5. Markov Decision Process for a Prosumer
  6. Consensus-update Actor-critic Algroithm
  7. Convergence Results
  8. Simulation Results
  9. Test Data
  10. Deep Neural Network Implementation
  11. Comaprison of Three Different MARL Algorithms
  12. Numerical Results
  13. Conclusion and Future Research

Key Result

Lemma 4.1

The supreumum in eq:payoff, $\sup_{\theta} J(\theta)$, exists and is finite.

Figures (9)

  • Figure 1: Market clearing using SDR
  • Figure 2: Social surplus comparison: total zero-marginal-cost supply less than total inflexible load
  • Figure 3: Social surplus: total zero-marginal-cost supply more than total inflexible load
  • Figure 4: Consensus MARL Framework for the Voltage Control with P2P Market
  • Figure 5: IEEE-13 test feeder
  • ...and 4 more figures

Theorems & Definitions (5)

  • Lemma 4.1
  • proof
  • Definition 4.1
  • Theorem 4.1
  • proof