Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cooperative Multi-agent RL with Communication Constraints

Nuoya Xiong, Aarti Singh

TL;DR

This work tackles cooperative multi-agent reinforcement learning under decentralized settings with limited communication by introducing base policy prediction (BPP), which proactively predicts a sequence of base policies to reduce the variance of importance sampling and extend the interval between communications. The authors first develop a novel BPP-enhanced policy gradient method for potential games, achieving $ ext{NE}$ with $ ilde{O}( ext{poly}( ext{max}_i| ext{A}_i|) ext{epsilon}^{-11/4})$ samples and $ ilde{O}( ext{epsilon}^{-3/4})$ communications, and then extend the framework to general Markov Cooperative Games using a PG oracle with two subroutines (PG-Share and V-Approx) to obtain agent-wise local maxima. Theoretical results show improved communication and sample complexity bounds compared to prior work, while experiments on potential games, congestion games, MAPPO, and SMAC demonstrate substantial reductions in communication rounds with competitive performance. The approach offers practical gains for large-scale decentralized MARL where communicating full state or reward information is costly, supporting more scalable cooperative learning in real-world systems. All mathematical claims are supported by rigorous proofs and the experiments include reproducible baselines and protocols.

Abstract

Cooperative MARL often assumes frequent access to global information in a data buffer, such as team rewards or other agents' actions, which is typically unrealistic in decentralized MARL systems due to high communication costs. When communication is limited, agents must rely on outdated information to estimate gradients and update their policies. A common approach to handle missing data is called importance sampling, in which we reweigh old data from a base policy to estimate gradients for the current policy. However, it quickly becomes unstable when the communication is limited (i.e. missing data probability is high), so that the base policy in importance sampling is outdated. To address this issue, we propose a technique called base policy prediction, which utilizes old gradients to predict the policy update and collect samples for a sequence of base policies, which reduces the gap between the base policy and the current policy. This approach enables effective learning with significantly fewer communication rounds, since the samples of predicted base policies could be collected within one communication round. Theoretically, we show that our algorithm converges to an $\varepsilon$-Nash equilibrium in potential games with only $O(\varepsilon^{-3/4})$ communication rounds and $O(poly(\max_i |A_i|)\varepsilon^{-11/4})$ samples, improving existing state-of-the-art results in communication cost, as well as sample complexity without the exponential dependence on the joint action space size. We also extend these results to general Markov Cooperative Games to find an agent-wise local maximum. Empirically, we test the base policy prediction algorithm in both simulated games and MAPPO for complex environments.

Cooperative Multi-agent RL with Communication Constraints

TL;DR

This work tackles cooperative multi-agent reinforcement learning under decentralized settings with limited communication by introducing base policy prediction (BPP), which proactively predicts a sequence of base policies to reduce the variance of importance sampling and extend the interval between communications. The authors first develop a novel BPP-enhanced policy gradient method for potential games, achieving with samples and communications, and then extend the framework to general Markov Cooperative Games using a PG oracle with two subroutines (PG-Share and V-Approx) to obtain agent-wise local maxima. Theoretical results show improved communication and sample complexity bounds compared to prior work, while experiments on potential games, congestion games, MAPPO, and SMAC demonstrate substantial reductions in communication rounds with competitive performance. The approach offers practical gains for large-scale decentralized MARL where communicating full state or reward information is costly, supporting more scalable cooperative learning in real-world systems. All mathematical claims are supported by rigorous proofs and the experiments include reproducible baselines and protocols.

Abstract

Cooperative MARL often assumes frequent access to global information in a data buffer, such as team rewards or other agents' actions, which is typically unrealistic in decentralized MARL systems due to high communication costs. When communication is limited, agents must rely on outdated information to estimate gradients and update their policies. A common approach to handle missing data is called importance sampling, in which we reweigh old data from a base policy to estimate gradients for the current policy. However, it quickly becomes unstable when the communication is limited (i.e. missing data probability is high), so that the base policy in importance sampling is outdated. To address this issue, we propose a technique called base policy prediction, which utilizes old gradients to predict the policy update and collect samples for a sequence of base policies, which reduces the gap between the base policy and the current policy. This approach enables effective learning with significantly fewer communication rounds, since the samples of predicted base policies could be collected within one communication round. Theoretically, we show that our algorithm converges to an -Nash equilibrium in potential games with only communication rounds and samples, improving existing state-of-the-art results in communication cost, as well as sample complexity without the exponential dependence on the joint action space size. We also extend these results to general Markov Cooperative Games to find an agent-wise local maximum. Empirically, we test the base policy prediction algorithm in both simulated games and MAPPO for complex environments.
Paper Structure (42 sections, 10 theorems, 113 equations, 4 figures, 1 table, 4 algorithms)

This paper contains 42 sections, 10 theorems, 113 equations, 4 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Markov Games
  4. Importance Sampling in RL
  5. Delayed Reward
  6. Preliminaries
  7. A Base Case: Potential Games
  8. Importance Sampling with Base Policy Prediction
  9. Base Policy Prediction
  10. Changing Condition
  11. Theoretical Results
  12. General MCG
  13. Experiments
  14. Potential Games
  15. Congestion Games
  16. ...and 27 more sections

Key Result

Theorem 4.2

Denote $M=\max\{\phi_{\max}, R_{\max}\} \ge 1$. Under Assumption assum:gap, following Algorithm alg: NE with learning rate $\eta = \frac{1}{2nM}$, if $\varepsilon \le \min\{1/n,1/4\}$, with probability at least $1-\delta$, policies $\{\hat{\pi}^t\}_{t \in [T]}$ satisfy that when $T=2/\varepsilon$. The total communication cost is bounded by $\mathcal{O}\left(\frac{n^{1/4}\phi_{\max}}{\varepsilon^{

Figures (4)

  • Figure 1: Convergence Result for Different Algorithms.
  • Figure 2: The Left two figures show the comparisons of base policy prediction under different communication intervals. The right two figures show the comparisons between MAPPO and base policy prediction.
  • Figure 3: Convergence results for SMAC environment
  • Figure 4: The Left two figures show the comparisons of base policy prediction under different communication intervals. The right two figures show the comparisons between IPPO and base policy prediction.

Theorems & Definitions (19)

  • Definition 3.1: Equilibrium Gap
  • Theorem 4.2
  • Corollary 4.3
  • Theorem 5.1
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • Lemma A.3
  • proof
  • ...and 9 more