On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation
Jiawei Huang, Batuhan Yardim, Niao He
TL;DR
This work examines the statistical efficiency of mean-field reinforcement learning under general function approximation by introducing the MF-MBED complexity measure, which captures the intrinsic difficulty of mean-field model classes. It develops maximal-likelihood estimation (MLE) based learning algorithms that achieve ε-optimal policies for MFC and ε-Nash equilibria for MFG with sample complexity polynomial in MF-MBED and under minimal assumptions of realizability and Lipschitz continuity. The paper provides concrete MF-MBED examples with low complexity, extends the framework to infinite model classes, and offers a rigorous proof-sketch connecting model prediction error to learning objectives while addressing density-dependent transitions. It also outlines open problems, including tighter bounds, computational considerations, and potential extensions to model-free methods in the mean-field setting, highlighting the practical impact for scalable, data-efficient multi-agent learning.
Abstract
In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an $ε$-optimal policy for MFC or an $ε$-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity.