Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Community-based Multi-Agent Reinforcement Learning with Transfer and Active Exploration

Zhaoyang Shi

TL;DR

The paper addresses coordination in multi-agent reinforcement learning (MARL) with heterogeneous agents by introducing a community-based MARL framework that uses latent overlapping communities and mixed memberships. It develops a two-time-scale actor-critic algorithm where each community maintains a shared $Q$-function and agents aggregate these estimates according to personalized membership weights, enabling transfer learning and active exploration with provable convergence under linear function approximation. A spectral MSCORE method estimates mixed memberships from observed interactions, enabling rapid transfer to new agents or tasks through $Q^{new}(s,a)=\sum_k \gamma^{new}(k) Q(s,a;\omega^{(k)})$. Active learning prioritizes uncertain communities to improve sample efficiency. Empirical results show improved performance over fixed-neighborhood MARL and support the theoretical convergence claims, highlighting the framework's scalability, transferability, and structured exploration benefits in coordinated multi-agent systems.

Abstract

We propose a new framework for multi-agent reinforcement learning (MARL), where the agents cooperate in a time-evolving network with latent community structures and mixed memberships. Unlike traditional neighbor-based or fixed interaction graphs, our community-based framework captures flexible and abstract coordination patterns by allowing each agent to belong to multiple overlapping communities. Each community maintains shared policy and value functions, which are aggregated by individual agents according to personalized membership weights. We also design actor-critic algorithms that exploit this structure: agents inherit community-level estimates for policy updates and value learning, enabling structured information sharing without requiring access to other agents' policies. Importantly, our approach supports both transfer learning by adapting to new agents or tasks via membership estimation, and active learning by prioritizing uncertain communities during exploration. Theoretically, we establish convergence guarantees under linear function approximation for both actor and critic updates. To our knowledge, this is the first MARL framework that integrates community structure, transferability, and active learning with provable guarantees.

Community-based Multi-Agent Reinforcement Learning with Transfer and Active Exploration

TL;DR

The paper addresses coordination in multi-agent reinforcement learning (MARL) with heterogeneous agents by introducing a community-based MARL framework that uses latent overlapping communities and mixed memberships. It develops a two-time-scale actor-critic algorithm where each community maintains a shared -function and agents aggregate these estimates according to personalized membership weights, enabling transfer learning and active exploration with provable convergence under linear function approximation. A spectral MSCORE method estimates mixed memberships from observed interactions, enabling rapid transfer to new agents or tasks through . Active learning prioritizes uncertain communities to improve sample efficiency. Empirical results show improved performance over fixed-neighborhood MARL and support the theoretical convergence claims, highlighting the framework's scalability, transferability, and structured exploration benefits in coordinated multi-agent systems.

Abstract

We propose a new framework for multi-agent reinforcement learning (MARL), where the agents cooperate in a time-evolving network with latent community structures and mixed memberships. Unlike traditional neighbor-based or fixed interaction graphs, our community-based framework captures flexible and abstract coordination patterns by allowing each agent to belong to multiple overlapping communities. Each community maintains shared policy and value functions, which are aggregated by individual agents according to personalized membership weights. We also design actor-critic algorithms that exploit this structure: agents inherit community-level estimates for policy updates and value learning, enabling structured information sharing without requiring access to other agents' policies. Importantly, our approach supports both transfer learning by adapting to new agents or tasks via membership estimation, and active learning by prioritizing uncertain communities during exploration. Theoretically, we establish convergence guarantees under linear function approximation for both actor and critic updates. To our knowledge, this is the first MARL framework that integrates community structure, transferability, and active learning with provable guarantees.
Paper Structure (20 sections, 5 theorems, 46 equations, 6 figures, 3 algorithms)

This paper contains 20 sections, 5 theorems, 46 equations, 6 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Notations
  3. Community-based Multi-agent Markov Decision Process (MDP)
  4. Agents with Mixed-memberships
  5. Objectives
  6. Actor-Critic (AC) for Agents with Mixed-membership
  7. Algorithms
  8. Mixed-membership Estimation
  9. Connection to Transfer and Active Learning
  10. Theoretical Results
  11. Numerical Results
  12. Discussion and Limitations
  13. Additional Algorithm on V-function
  14. Proof
  15. Proof of Theorem \ref{['thm:convergen_omega']}
  16. ...and 5 more sections

Key Result

Proposition 3.1

For any $\theta\in\Theta$, recall that $\pi_{\theta}$ is the policy and $J(\theta)$ means the globally averaged return given in J. $Q_{\theta}$ and $A_{\theta}$ represent the corresponding action-value function and advantage function respectively. Moreover, we define the local advantage function $A_ where $\widetilde{V}_{\theta}^i(s,a^{-i})=\sum_{a^i\in\mathcal{A}^i}\pi_{\theta^i}^i(s,a^i)Q_{\thet

Figures (6)

  • Figure 1: Values of the community wide parameter $\omega_{t}^{(k)}$. In this experiment, we consider $\omega_{t}^{(k)}\in\mathbb{R}^{10}$, i.e., $m_{\Omega}=10$. Due to space limitation, we only present the first two dimensions of each $\omega_{t}^{(k)}$. The complete details are included in Section \ref{['sec:exp_detail']} in the appendix.
  • Figure 2: Values of the agent policy parameter $\theta_{t}^i$. In this experiment, we consider $N=20$ agents. Due to space limitation, we only present some of their $\theta_{t}^i$. The complete details are included in Section \ref{['sec:exp_detail']} in the appendix.
  • Figure 3: Values of globally averaged return $J(\theta_t)$ output by community-based MARL (red) and neighbor-based MARL (blue).
  • Figure 4: Values of the community wide parameter $\omega_{t}^{(k)}$. In this experiment, we consider $\omega_{t}^{(k)}\in\mathbb{R}^{10}$, i.e., $m_{\Omega}=10$.
  • Figure 5: Values of the agent policy parameter $\theta_{t}^i$ for $i=1,\ldots,10$. In this experiment, we consider $N=20$ agents. It shows the first half of the agents.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Proposition 3.1: Theorem 3.1 in zhang2018fully
  • Theorem 5.1
  • Theorem 5.2
  • Theorem C.1
  • Theorem C.2