Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bi-level Mean Field: Dynamic Grouping for Large-Scale MARL

Yuxuan Zheng, Yihe Zhou, Feiyang Xu, Mingli Song, Shunyu Liu

TL;DR

Large-scale MARL faces the curse of dimensionality and aggregation noise when using naïve mean-field (MF) approximations. The authors propose Bi-level Mean Field (BMF), which combines dynamic grouping via a VAE-based encoder with a bi-level interaction mechanism that separately handles intra-group MF and inter-group attention, reducing aggregation noise while maintaining scalability. They provide theoretical analysis showing a bounded approximation error under MF assumptions and demonstrate across Firefighter, Adversarial Pursuit, and Battle that BMF achieves superior or competitive performance with lower computational cost than state-of-the-art MF variants and GAT-MF, including zero-shot generalization to different agent counts. The work advances scalable, adaptive neighbor modeling in large-scale MARL and suggests future avenues for integrating adaptive grouping with BMF in broader practical applications.

Abstract

Large-scale Multi-Agent Reinforcement Learning (MARL) often suffers from the curse of dimensionality, as the exponential growth in agent interactions significantly increases computational complexity and impedes learning efficiency. To mitigate this, existing efforts that rely on Mean Field (MF) simplify the interaction landscape by approximating neighboring agents as a single mean agent, thus reducing overall complexity to pairwise interactions. However, these MF methods inevitably fail to account for individual differences, leading to aggregation noise caused by inaccurate iterative updates during MF learning. In this paper, we propose a Bi-level Mean Field (BMF) method to capture agent diversity with dynamic grouping in large-scale MARL, which can alleviate aggregation noise via bi-level interaction. Specifically, BMF introduces a dynamic group assignment module, which employs a Variational AutoEncoder (VAE) to learn the representations of agents, facilitating their dynamic grouping over time. Furthermore, we propose a bi-level interaction module to model both inter- and intra-group interactions for effective neighboring aggregation. Experiments across various tasks demonstrate that the proposed BMF yields results superior to the state-of-the-art methods.

Bi-level Mean Field: Dynamic Grouping for Large-Scale MARL

TL;DR

Large-scale MARL faces the curse of dimensionality and aggregation noise when using naïve mean-field (MF) approximations. The authors propose Bi-level Mean Field (BMF), which combines dynamic grouping via a VAE-based encoder with a bi-level interaction mechanism that separately handles intra-group MF and inter-group attention, reducing aggregation noise while maintaining scalability. They provide theoretical analysis showing a bounded approximation error under MF assumptions and demonstrate across Firefighter, Adversarial Pursuit, and Battle that BMF achieves superior or competitive performance with lower computational cost than state-of-the-art MF variants and GAT-MF, including zero-shot generalization to different agent counts. The work advances scalable, adaptive neighbor modeling in large-scale MARL and suggests future avenues for integrating adaptive grouping with BMF in broader practical applications.

Abstract

Large-scale Multi-Agent Reinforcement Learning (MARL) often suffers from the curse of dimensionality, as the exponential growth in agent interactions significantly increases computational complexity and impedes learning efficiency. To mitigate this, existing efforts that rely on Mean Field (MF) simplify the interaction landscape by approximating neighboring agents as a single mean agent, thus reducing overall complexity to pairwise interactions. However, these MF methods inevitably fail to account for individual differences, leading to aggregation noise caused by inaccurate iterative updates during MF learning. In this paper, we propose a Bi-level Mean Field (BMF) method to capture agent diversity with dynamic grouping in large-scale MARL, which can alleviate aggregation noise via bi-level interaction. Specifically, BMF introduces a dynamic group assignment module, which employs a Variational AutoEncoder (VAE) to learn the representations of agents, facilitating their dynamic grouping over time. Furthermore, we propose a bi-level interaction module to model both inter- and intra-group interactions for effective neighboring aggregation. Experiments across various tasks demonstrate that the proposed BMF yields results superior to the state-of-the-art methods.
Paper Structure (18 sections, 1 theorem, 19 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 18 sections, 1 theorem, 19 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Related work
  2. Preliminary
  3. Multi-agent Markov Decision Process
  4. Q Learning
  5. Policy Gradient
  6. Method
  7. Overview
  8. Dynamic Group Assignment
  9. Bi-level Mean Field
  10. Theoretical Analysis
  11. Experiments
  12. Settings
  13. Performance
  14. Computational Efficiency
  15. Zero-Shot Generalization
  16. ...and 3 more sections

Key Result

Theorem 1

When considering agent $j$ in group $m$, the global Q-function can be represented as: where $\tilde{a}_j$ is intra-group MF action and $\tilde{a}_m$ is the inter-group MF action.

Figures (5)

  • Figure 1: Normal, mean field and bi-level mean field methods apply to the large-scale MARL scenario. Limitation of normal large-scale MARL is the excessively high dimension of the concatenated critic input. Limitation of mean field is ignoring agents' difference and solely using average aggregation, resulting in a loss of precision. Our proposed Bi-level Mean Field (BMF) effectively reduces the critic input dimension while ensuring precision.
  • Figure 2: The Bi-level Mean Field (BMF) framework. The group assignment forward model for learning agent representations is based on a VAE architecture. The intra-group MF considers the influence of same-type agents and approximates the average effect for a group. The inter-group MF focuses on the differences between different groups and calculates the total Q value.
  • Figure 3: Compare BMF with other methods in various large-scale MARL environments. All experimental results are illustrated with the mean and the standard deviation of the metrics over 4 random seeds for a fair comparison.
  • Figure 4: Ablations about group assignment and the number of clusters under the Battle environment.
  • Figure 5: Visualization of agents collaboration and competition under the Battle scenario.

Theorems & Definitions (2)

  • Theorem 1: BMF approximation
  • proof