Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Communication-Efficient Decentralized Actor-Critic Algorithm

Xiaoxing Ren, Nicola Bastianello, Thomas Parisini, Andreas A. Malikopoulos

TL;DR

This work addresses learning in multi-agent systems with limited communication by developing a communication-efficient decentralized actor-critic algorithm that uses local training via LT-ADMM and Markovian mini-batches, while employing a neural-network critic. It proves finite-time convergence under Markovian sampling, with sample complexity $O( ext{eps}^{-3})$ and communication complexity $O( ext{eps}^{-1} au^{-1})$, and shows the final error depends on the neural-approximation gap $ ext{varsigma}_{ ext{approx}}$. The method maintains consensus through bridge variables and achieves reliable performance with a single communication round per outer iteration, validated by cooperative navigation experiments. The results offer a practical, scalable approach for efficient MARL with nonlinear function approximation and provable guarantees, highlighting favorable trade-offs between local computation and inter-agent communication.

Abstract

In this paper, we study the problem of reinforcement learning in multi-agent systems where communication among agents is limited. We develop a decentralized actor-critic learning framework in which each agent performs several local updates of its policy and value function, where the latter is approximated by a multi-layer neural network, before exchanging information with its neighbors. This local training strategy substantially reduces the communication burden while maintaining coordination across the network. We establish finite-time convergence analysis for the algorithm under Markov-sampling. Specifically, to attain the $\varepsilon$-accurate stationary point, the sample complexity is of order $\mathcal{O}(\varepsilon^{-3})$ and the communication complexity is of order $\mathcal{O}(\varepsilon^{-1}τ^{-1})$, where tau denotes the number of local training steps. We also show how the final error bound depends on the neural network's approximation quality. Numerical experiments in a cooperative control setting illustrate and validate the theoretical findings.

A Communication-Efficient Decentralized Actor-Critic Algorithm

TL;DR

This work addresses learning in multi-agent systems with limited communication by developing a communication-efficient decentralized actor-critic algorithm that uses local training via LT-ADMM and Markovian mini-batches, while employing a neural-network critic. It proves finite-time convergence under Markovian sampling, with sample complexity and communication complexity , and shows the final error depends on the neural-approximation gap . The method maintains consensus through bridge variables and achieves reliable performance with a single communication round per outer iteration, validated by cooperative navigation experiments. The results offer a practical, scalable approach for efficient MARL with nonlinear function approximation and provable guarantees, highlighting favorable trade-offs between local computation and inter-agent communication.

Abstract

In this paper, we study the problem of reinforcement learning in multi-agent systems where communication among agents is limited. We develop a decentralized actor-critic learning framework in which each agent performs several local updates of its policy and value function, where the latter is approximated by a multi-layer neural network, before exchanging information with its neighbors. This local training strategy substantially reduces the communication burden while maintaining coordination across the network. We establish finite-time convergence analysis for the algorithm under Markov-sampling. Specifically, to attain the -accurate stationary point, the sample complexity is of order and the communication complexity is of order , where tau denotes the number of local training steps. We also show how the final error bound depends on the neural network's approximation quality. Numerical experiments in a cooperative control setting illustrate and validate the theoretical findings.
Paper Structure (28 sections, 8 theorems, 85 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 28 sections, 8 theorems, 85 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background on Reinforcement Learning
  3. Decentralized Actor--Critic Learning
  4. Overview of This Work
  5. Comparison with the State of the Art
  6. Problem Formulation
  7. State and action
  8. State transition probability
  9. Observations and policy
  10. Reward function
  11. System Model and Policy Structure
  12. Decentralized optimization setup
  13. Algorithm Design
  14. Nonlinear approximation of the value function
  15. Proposed algorithm
  16. ...and 13 more sections

Key Result

Theorem 1

Let Assumptions as:graph, ass:r_bounded, ass:ergodicity, ass:s_bounded, ass:sigma_smooth, ass:x_bounded, ass:initial_bounded, and assu:smoothness hold. Choose learning rates that satisfy $\alpha \leq \bar{\alpha}$ and $\frac{1}{\tau\lambda_u\rho} \leq \beta < \frac{2}{\tau\lambda_u\rho}$, where $\b where $\bar{\omega}_k = \frac{1}{N} \sum_{i = 1}^N {\omega}^i_k$, $\lambda_l$ is the smallest nonze

Figures (6)

  • Figure 1: Comparison with related algorithms. The x-axis indicates the number of communication rounds.
  • Figure 2: Plots of the consensus error and critic loss of Algorithm \ref{['alg: LT-ADMM Actor-Critic']}.
  • Figure 3: Trajectories of five agents using the policy learned by Algorithm \ref{['alg: LT-ADMM Actor-Critic']}, case 1.
  • Figure 4: Trajectories of five agents using the policy learned by Algorithm \ref{['alg: LT-ADMM Actor-Critic']}, case 2.
  • Figure 5: Trajectories of five agents using the policy learned by Algorithm \ref{['alg: LT-ADMM Actor-Critic']}, case 3.
  • ...and 1 more figures

Theorems & Definitions (17)

  • Remark 1
  • Remark 2
  • Theorem 1
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • ...and 7 more