Learning Large-Scale Competitive Team Behaviors with Mean-Field Interactions and Online Opponent Modeling

TL;DR

This work tackles the scalability challenge of multi-agent reinforcement learning in large-scale competitive teams by marrying mean-field theory with proximal policy optimization. It introduces MF-MAPPO, a PPO-based algorithm with a shared actor per team and a minimally-informed critic dependent on mean-field inputs, enabling training on finite-population simulators and extending to partial observability via gradient-regularized training. A decentralized mean-field estimator, Dynamic-Projected Consensus (D-PC), and a gradient-regularized extension (GR-MF-MAPPO) address partial observability and limited communication, with theoretical guarantees on gradient convergence and regret bounds. Empirical results on the MFEnv benchmarks demonstrate superior performance, rich heterogeneous behaviors, and robust operation under communication constraints, highlighting the practical impact for large-scale competitive MARL in realistic scenarios.

Abstract

While multi-agent reinforcement learning (MARL) has been proven effective across both collaborative and competitive tasks, existing algorithms often struggle to scale to large populations of agents. Recent advancements in mean-field (MF) theory provide scalable solutions by approximating population interactions as a continuum, yet most existing frameworks focus exclusively on either fully cooperative or purely competitive settings. To bridge this gap, we introduce MF-MAPPO, a mean-field extension of PPO designed for zero-sum team games that integrate intra-team cooperation with inter-team competition. MF-MAPPO employs a shared actor and a minimally informed critic per team and is trained directly on finite-population simulators, thereby enabling deployment to realistic scenarios with thousands of agents. We further show that MF-MAPPO naturally extends to partially observable settings through a simple gradient-regularized training scheme. Our evaluation utilizes large-scale benchmark scenarios using our own testing simulation platform for MF team games (MFEnv), including offense-defense battlefield tasks as well as variants of population-based rock-paper-scissors games that admit analytical solutions, for benchmarking. Across these benchmarks, MF-MAPPO outperforms existing methods and exhibits complex, heterogeneous behaviors, demonstrating the effectiveness of combining mean-field theory and MARL techniques at scale.

Figures (25)

• Figure 1: (a) Battlefield as a ZS-MFTG (b) Overview of the architecture of MF-MAPPO.
• Figure 2: (a) 150 initializations of $\mu_{t=0} = [1, 0, 0]^{\hbox{\tiny\sf T}}$ and $\nu_{t=0} = [0, 1, 0]^{\hbox{\tiny\sf T}}$ for cRPS; $N_1=N_2=1,000$ (b) Deploying MF-MAPPO trained on $N_1=N_2=1,000$ to varying team sizes.
• Figure 3: I. Training curve for Battlefield on a 4x4 grid (Blue team); II. Example configuration; III. Comparing $\mathrm{d}_{\mathrm{TV}} ( \cdot )$ for D-PC and Benchmark estimator for different $R_\textrm{com}$.
• Figure 4: I. Red is concentrated; 30% Blue are at the bottom, rest are at the top II. Blue is evenly split, Red is concentrated III. Comparing $\mathrm{d}_{\mathrm{TV}} ( \cdot )$ for D-PC and the Benchmark estimator for different $R_{\textrm{com}}$ IV. % error in cumulative rewards under varying communication bandwidths V. % error in cumulative rewards under noisy communication $R_{\textrm{com}}=7.$
• Figure 5: Example configurations of SIS-Epidemiology ZS-MFTG; green cell is the hospital.

Theorems & Definitions (43)

• Definition 1
• Definition 2: Identical team policy
• Theorem 1
• Proposition 1
• Theorem 2
• Theorem 3
• Proposition 2
• Proposition 3
• Theorem 4
• Definition 3: Visibility Graph