Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networks

Amin Shojaeighadikolaei, Zsolt Talata, Morteza Hashemi

TL;DR

The paper addresses demand-side management for EV charging under real-time pricing by formulating the problem as Dec-POMDP and evaluating two MARL variants: CTDE-DDPG and I-DDPG. It shows that centralized training with a shared critic (CTDE) promotes cooperative learning in a distributed execution setting, yielding smoother charging, reduced price volatility, and improved fairness, albeit with higher policy-gradient variance. Theoretical analysis proves equal expected policy gradients between CTDE-DDPG and I-DDPG at convergence, while CTDE-DDPG incurs larger variance, a trade-off that favors CTDE in dynamic grid conditions. Numerical experiments on an IEEE-like 5-bus system with up to 20 agents demonstrate that CTDE-DDPG reduces total variation by around 36% and charging costs by about 9.1% on average, validating the benefit of cooperation for scalable, privacy-preserving EV charging control. Overall, the work highlights the practical impact of CTDE-based MARL for robust, cost-efficient DSM in modern smart grids.

Abstract

The widespread adoption of electric vehicles (EVs) poses several challenges to power distribution networks and smart grid infrastructure due to the possibility of significantly increasing electricity demands, especially during peak hours. Furthermore, when EVs participate in demand-side management programs, charging expenses can be reduced by using optimal charging control policies that fully utilize real-time pricing schemes. However, devising optimal charging methods and control strategies for EVs is challenging due to various stochastic and uncertain environmental factors. Currently, most EV charging controllers operate based on a centralized model. In this paper, we introduce a novel approach for distributed and cooperative charging strategy using a Multi-Agent Reinforcement Learning (MARL) framework. Our method is built upon the Deep Deterministic Policy Gradient (DDPG) algorithm for a group of EVs in a residential community, where all EVs are connected to a shared transformer. This method, referred to as CTDE-DDPG, adopts a Centralized Training Decentralized Execution (CTDE) approach to establish cooperation between agents during the training phase, while ensuring a distributed and privacy-preserving operation during execution. We theoretically examine the performance of centralized and decentralized critics for the DDPG-based MARL implementation and demonstrate their trade-offs. Furthermore, we numerically explore the efficiency, scalability, and performance of centralized and decentralized critics. Our theoretical and numerical results indicate that, despite higher policy gradient variances and training complexity, the CTDE-DDPG framework significantly improves charging efficiency by reducing total variation by approximately %36 and charging cost by around %9.1 on average...

Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networks

TL;DR

The paper addresses demand-side management for EV charging under real-time pricing by formulating the problem as Dec-POMDP and evaluating two MARL variants: CTDE-DDPG and I-DDPG. It shows that centralized training with a shared critic (CTDE) promotes cooperative learning in a distributed execution setting, yielding smoother charging, reduced price volatility, and improved fairness, albeit with higher policy-gradient variance. Theoretical analysis proves equal expected policy gradients between CTDE-DDPG and I-DDPG at convergence, while CTDE-DDPG incurs larger variance, a trade-off that favors CTDE in dynamic grid conditions. Numerical experiments on an IEEE-like 5-bus system with up to 20 agents demonstrate that CTDE-DDPG reduces total variation by around 36% and charging costs by about 9.1% on average, validating the benefit of cooperation for scalable, privacy-preserving EV charging control. Overall, the work highlights the practical impact of CTDE-based MARL for robust, cost-efficient DSM in modern smart grids.

Abstract

The widespread adoption of electric vehicles (EVs) poses several challenges to power distribution networks and smart grid infrastructure due to the possibility of significantly increasing electricity demands, especially during peak hours. Furthermore, when EVs participate in demand-side management programs, charging expenses can be reduced by using optimal charging control policies that fully utilize real-time pricing schemes. However, devising optimal charging methods and control strategies for EVs is challenging due to various stochastic and uncertain environmental factors. Currently, most EV charging controllers operate based on a centralized model. In this paper, we introduce a novel approach for distributed and cooperative charging strategy using a Multi-Agent Reinforcement Learning (MARL) framework. Our method is built upon the Deep Deterministic Policy Gradient (DDPG) algorithm for a group of EVs in a residential community, where all EVs are connected to a shared transformer. This method, referred to as CTDE-DDPG, adopts a Centralized Training Decentralized Execution (CTDE) approach to establish cooperation between agents during the training phase, while ensuring a distributed and privacy-preserving operation during execution. We theoretically examine the performance of centralized and decentralized critics for the DDPG-based MARL implementation and demonstrate their trade-offs. Furthermore, we numerically explore the efficiency, scalability, and performance of centralized and decentralized critics. Our theoretical and numerical results indicate that, despite higher policy gradient variances and training complexity, the CTDE-DDPG framework significantly improves charging efficiency by reducing total variation by approximately %36 and charging cost by around %9.1 on average...
Paper Structure (18 sections, 3 theorems, 22 equations, 9 figures, 1 table)

This paper contains 18 sections, 3 theorems, 22 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Model and Problem Formulation
  4. System Model
  5. Centralized EV Network Optimization
  6. Distributed EV Network Optimization
  7. Principles of the algorithm
  8. Policy Gradient Method
  9. Agent Setup for EV Network
  10. Multi-Agent Control Strategy
  11. Multi-Agent Methods
  12. Analysis of Multi-Agent DDPG Methods
  13. Numerical Results
  14. Experimental Setting
  15. MARL General Performance
  16. ...and 3 more sections

Key Result

Lemma 1

After the convergence of the critic network for CTDE-DDPG and I-DDPG, the following equality holds:

Figures (9)

  • Figure 1: Electric vehicle charging network with a shared energy source.
  • Figure 2: Centralized training decentralized execution (CTDE) multi-agent reinforcement learning framework for EV charging network control.
  • Figure 3: IEEE 5-bus testbed system. An example of how the proposed EV charging network can be integrated into a distribution system.
  • Figure 4: Performance comparison for CTDE-DDPG and I-DDPG frameworks in terms of average battery level during the charging period for 10 agent scenarios.
  • Figure 5: Charging behavior comparison between CTDE-DDPG and I-DDPG frameworks in terms of average charging rate over the charging period for 10 agent scenarios, where one denotes the full rate charging.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof