Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning
Batuhan Yardim, Niao He
TL;DR
The paper tackles learning in large-scale MARL under approximate symmetry when a known MFG model is unavailable. It introduces an induced mean-field game constructed from any finite N-player dynamic game via Kirszbraun Lipschitz extensions and defines α,β-symmetric DGs to quantify heterogeneity. It proves that the induced MFG Nash equilibrium provides an approximate equilibrium for the original game with explicit bounds, and establishes TD-learning guarantees along with a monotone PMD-based method achieving an ε-NE with sample complexity $ ilde{O}(oldsymbol{}^{-6})$ trajectories. Empirical validation on benchmarks with thousands of agents demonstrates scalability and efficiency gains from symmetrized neural policies and end-to-end learning without explicit MFG models.
Abstract
Mean-field games (MFG) have become significant tools for solving large-scale multi-agent reinforcement learning problems under symmetry. However, the assumption of exact symmetry limits the applicability of MFGs, as real-world scenarios often feature inherent heterogeneity. Furthermore, most works on MFG assume access to a known MFG model, which might not be readily available for real-world finite-agent games. In this work, we broaden the applicability of MFGs by providing a methodology to extend any finite-player, possibly asymmetric, game to an "induced MFG". First, we prove that $N$-player dynamic games can be symmetrized and smoothly extended to the infinite-player continuum via explicit Kirszbraun extensions. Next, we propose the notion of $α,β$-symmetric games, a new class of dynamic population games that incorporate approximate permutation invariance. For $α,β$-symmetric games, we establish explicit approximation bounds, demonstrating that a Nash policy of the induced MFG is an approximate Nash of the $N$-player dynamic game. We show that TD learning converges up to a small bias using trajectories of the $N$-player game with finite-sample guarantees, permitting symmetrized learning without building an explicit MFG model. Finally, for certain games satisfying monotonicity, we prove a sample complexity of $\widetilde{\mathcal{O}}(\varepsilon^{-6})$ for the $N$-agent game to learn an $\varepsilon$-Nash up to symmetrization bias. Our theory is supported by evaluations on MARL benchmarks with thousands of agents.