Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

Pengcheng Dai, He Wang, Dongming Wang, Wenwu Yu

TL;DR

This work addresses constrained multi-agent RL with tightly coupled policies and privacy constraints. It introduces a distributed primal-dual framework (DSPD) where agents exchange true policy parameters and dual variables only over a time-varying learning network, and compute approximated policy gradients using information from their κ+-hop neighborhoods. Theoretical guarantees show ε-first-order stationary convergence with an approximation error that scales as O($\gamma^{(κ+1)/κ_p}$), and the parameter estimates of other agents converge with rate O($1/m$). Empirical validation in GridWorld demonstrates faster convergence and constraint satisfaction compared to independent-policy baselines, highlighting the practical viability and security benefits of the proposed approach.

Abstract

In this work, we investigate constrained multi-agent reinforcement learning (CMARL), where agents collaboratively maximize the sum of their local objectives while satisfying individual safety constraints. We propose a framework where agents adopt coupled policies that depend on both local states and parameters, as well as those of their $κ_p$-hop neighbors, with $κ_p>0$ denoting the coupling distance. A distributed primal-dual algorithm is further developed under this framework, wherein each agent has access only to state-action pairs within its $2κ_p$-hop neighborhood and to reward information within its $κ+ 2κ_p$-hop neighborhood, with $κ> 0$ representing the truncation distance. Moreover, agents are not permitted to directly share their true policy parameters or Lagrange multipliers. Instead, each agent constructs and maintains local estimates of these variables for other agents and employs such estimates to execute its policy. Additionally, these estimates are further updated and exchanged exclusively through an independent, time-varying networks, which enhances the overall system security. We establish that, with high probability, our algorithm can achieve an $ε$-first-order stationary convergence with an approximation error of $\mathcal{O}(γ^{\frac{κ+1}{κ_{p}}})$ for discount factor $γ\in(0,1)$. Finally, simulations in GridWorld environment are conducted to demonstrate the effectiveness of the proposed algorithm.

Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

TL;DR

This work addresses constrained multi-agent RL with tightly coupled policies and privacy constraints. It introduces a distributed primal-dual framework (DSPD) where agents exchange true policy parameters and dual variables only over a time-varying learning network, and compute approximated policy gradients using information from their κ+-hop neighborhoods. Theoretical guarantees show ε-first-order stationary convergence with an approximation error that scales as O(), and the parameter estimates of other agents converge with rate O(). Empirical validation in GridWorld demonstrates faster convergence and constraint satisfaction compared to independent-policy baselines, highlighting the practical viability and security benefits of the proposed approach.

Abstract

In this work, we investigate constrained multi-agent reinforcement learning (CMARL), where agents collaboratively maximize the sum of their local objectives while satisfying individual safety constraints. We propose a framework where agents adopt coupled policies that depend on both local states and parameters, as well as those of their -hop neighbors, with denoting the coupling distance. A distributed primal-dual algorithm is further developed under this framework, wherein each agent has access only to state-action pairs within its -hop neighborhood and to reward information within its -hop neighborhood, with representing the truncation distance. Moreover, agents are not permitted to directly share their true policy parameters or Lagrange multipliers. Instead, each agent constructs and maintains local estimates of these variables for other agents and employs such estimates to execute its policy. Additionally, these estimates are further updated and exchanged exclusively through an independent, time-varying networks, which enhances the overall system security. We establish that, with high probability, our algorithm can achieve an -first-order stationary convergence with an approximation error of for discount factor . Finally, simulations in GridWorld environment are conducted to demonstrate the effectiveness of the proposed algorithm.

Paper Structure

This paper contains 31 sections, 12 theorems, 111 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. CMARL problem under coupled policies
  4. Formulation of the CMARL problem under coupled policies
  5. Approximate first-order stationary point
  6. Coupled policy gradient
  7. DSPD algorithm
  8. Approximated policy gradient
  9. Estimation of private parameters and principles of interaction
  10. DSPD algorithm designing
  11. Convergence Analysis
  12. Main results
  13. Analysis of Theorem \ref{['thelemmaofapproximatedpolicygradient']}
  14. Analysis of Theorem \ref{['thetheoremconvergenceofpolicygradient']}
  15. Simulation
  16. ...and 16 more sections

Key Result

Lemma 1

In CMARL problem (theproblemofCMARL), for any joint policy $\bm{\pi_{\theta}}$, the gradients of $\mathcal{L}(\bm{\theta},\bm{\mu})$ with respect to $\theta_{i}$ and $\mu_{i}$ are respectively represented as and

Figures (3)

  • Figure 1: Performance of Algorithm 1 and SPDAC on the objective function.
  • Figure 2: Constraint performance of Algorithm 1 and SPDAC.
  • Figure 3: The evolution of the estimation errors for agents with respect to the Lagrange multipliers and policy parameters of other agents.

Theorems & Definitions (26)

  • Remark 1
  • Definition 1
  • Lemma 1
  • Remark 2
  • Theorem 1: Approximated policy gradient error
  • Theorem 2: Convergence of parameter estimations
  • Theorem 3: $\epsilon$-first-order stationary convergence
  • Lemma 2
  • Lemma 3
  • Lemma 4: Unbiased gradient estimates for (\ref{['theapproximatedgradientinalgorithmdesignmu']}) and (\ref{['theapproximatedgradientinalgorithmdesign']})
  • ...and 16 more