Learning Mean Field Control on Sparse Graphs
Christian Fabian, Kai Cui, Heinz Koeppl
TL;DR
This work tackles cooperative MARL on extremely sparse graphs where the average degree remains finite, a regime where graphon-based mean-field models underperform. It introduces Locally Weak Mean Field Control (LWMFC), leveraging local weak convergence to define degree-specific mean fields and a limiting system, together with a two-systems approximation to manage high-degree neighborhoods. The authors prove MF and objective convergence and develop scalable learning algorithms, including a limiting MFC MDP solver and a MARL method that operates directly on real networks. Empirical results on synthetic Chung-Lu graphs and eight real networks demonstrate that LWMFC and its extensive variant outperform existing mean-field approaches, providing practical tools for scalable policy learning in sparse networks with diverging degree variance.
Abstract
Large agent networks are abundant in applications and nature and pose difficult challenges in the field of multi-agent reinforcement learning (MARL) due to their computational and theoretical complexity. While graphon mean field games and their extensions provide efficient learning algorithms for dense and moderately sparse agent networks, the case of realistic sparser graphs remains largely unsolved. Thus, we propose a novel mean field control model inspired by local weak convergence to include sparse graphs such as power law networks with coefficients above two. Besides a theoretical analysis, we design scalable learning algorithms which apply to the challenging class of graph sequences with finite first moment. We compare our model and algorithms for various examples on synthetic and real world networks with mean field algorithms based on Lp graphons and graphexes. As it turns out, our approach outperforms existing methods in many examples and on various networks due to the special design aiming at an important, but so far hard to solve class of MARL problems.