Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Deep Reinforcement Learning Under Constrained Communications

Shahil Shaik, Jonathon M. Smereka, Yue Wang

TL;DR

This work addresses the limitations of centralized training in multi-agent reinforcement learning by proposing a fully distributed framework that relies solely on local observations and peer-to-peer communication. It introduces Distributed Graph Attention Networks (D-GAT) for multi-hop global state inference and DG-MAPPO, a PPO-like distributed MARL algorithm that uses locally learned critics and shared rewards. The approach matches or exceeds CTDE baselines on diverse cooperative benchmarks (SMAC, Google Football, MA-MuJoCo), demonstrating robust coordination with sparse communication and scalability to large teams. By eliminating the need for privileged global information, the method enables deployment-ready, resilient MARL in dynamic, real-world networks.

Abstract

Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.

Multi-Agent Deep Reinforcement Learning Under Constrained Communications

TL;DR

This work addresses the limitations of centralized training in multi-agent reinforcement learning by proposing a fully distributed framework that relies solely on local observations and peer-to-peer communication. It introduces Distributed Graph Attention Networks (D-GAT) for multi-hop global state inference and DG-MAPPO, a PPO-like distributed MARL algorithm that uses locally learned critics and shared rewards. The approach matches or exceeds CTDE baselines on diverse cooperative benchmarks (SMAC, Google Football, MA-MuJoCo), demonstrating robust coordination with sparse communication and scalability to large teams. By eliminating the need for privileged global information, the method enables deployment-ready, resilient MARL in dynamic, real-world networks.

Abstract

Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.
Paper Structure (31 sections, 1 theorem, 29 equations, 11 figures, 5 tables, 1 algorithm)

This paper contains 31 sections, 1 theorem, 29 equations, 11 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Graph-Attention Networks
  5. Policy Gradient Theorem for Multi-Agent Reinforcement Learning
  6. Method
  7. Distributed Graph Attention Networks
  8. Distributed Graph-Attention MAPPO
  9. Results
  10. Experiment Setup
  11. Performance on Cooperative Benchmarks
  12. Conclusion
  13. Appendix
  14. Use of LLMs
  15. Pseudo Code for DG-MAPPO
  16. ...and 16 more sections

Key Result

Theorem 1

Consider $N$ agents with local policies $\pi^i_{\theta^i}$ parameterized by $\theta^i$, and let the joint policy be $\pi_\theta = \prod_{i=1}^N \pi_{\theta^i}^i, {\theta}=[\theta^1,\dots,\theta^n]^\top$. The collective objective is to maximize the globally averaged return $J({\pi_\theta})$ defined i where $A_{{\theta}}(\boldsymbol{o}_t,\boldsymbol{a}_t) = R(\boldsymbol{o}_t,\boldsymbol{a}_t) + \ga

Figures (11)

  • Figure 1: Illustration of the forward and backward passes in the proposed D-GAT framework. 1) Forward pass (top, red): At each layer $k$, node $4$ aggregates information from its neighbors $\{h^j_k\|h^4_k\}_{j\in\mathcal{N}^4}$ by applying a linear transformation, Softplus normalization, summation-based aggregation, and a nonlinearity $\sigma$ to produce the updated embedding $\hat{\boldsymbol{h}}^4_{k+1}$. This process is repeated for $k=0,\dots,K-1$, where $K$ is the predefined number of hops. 2) Backward pass (bottom, blue): Node $5$ updates its local D-GAT parameters $\boldsymbol{\psi}^5$ via decentralized stochastic gradient descent (D-SGD). First, a local step computes the half-step parameters $\boldsymbol{\psi}^5_{t+\frac{1}{2}}$ using the local update step of Equation (\ref{['eq:D-SGD']}). Next, a neighbor averaging step mixes parameters from neighbors via the neighbor averaging step of Equation (\ref{['eq:D-SGD']}) to get the updated D-GAT parameters $\boldsymbol{\psi}^5_{t}$, enabling distributed training without a central coordinator.
  • Figure 2: DG-MAPPO Framework. Each agent receives raw local observations $\boldsymbol{o}^i_t$ from the environment. Agents then communicate with neighbors using D-GAT to get global state inference $\hat{\boldsymbol{o}}^i_t$ which is then concatenated with raw local observations to get $\tilde{\boldsymbol{o}}^i_t=[\boldsymbol{o}^i_t, \hat{\boldsymbol{o}}^i_t]$. This combined observation is used by both local policies $\pi^i_{\theta^i}$ to generate actions and by value functions $V^i_{\phi^i}$ to estimate global returns. PPO performs per-agent policy updates using the advantage estimates derived from GAE.
  • Figure 3: Evaluation performance of DG-MAPPO compared to the CTDE baselines. (a) Results on the Google Research Football Academy 3 vs 1 with Keeper scenario. (b) Results on the Multi-Agent MuJoCo Multi-HalfCheetah (6×1) task.
  • Figure 4: Comparison of communication cost for CTDE and DG-MAPPO as team size N increases. All methods assume identical per-message and per-observation sizes. The DG-MAPPO curve uses a hop budget of N/2, consistent with our ablation findings on effective message-passing depth. The multi-hop CTDE baseline models a worst-case 1-D network topology. Results are shown on a log scale to highlight differences in growth rates.
  • Figure 5: Evaluation win rate of DG-MAPPO under different hop counts with clipped sight range in the SMAC environment.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Remark 1
  • Theorem 1: Policy Gradient for MARL zhang2018fully
  • Definition 1: Distributed MARL