Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions

Tobias Schmidt, Kai Cui

TL;DR

The paper addresses the curse of dimensionality in multi-agent control on sparse networks by replacing the classical mean-field state with a distribution over decorated rooted neighborhoods, $\boldsymbol{\mu} \in \mathcal{P}(\mathcal{G}_*^{\mathcal{X}})$. It proves a horizon-dependent locality theorem: for a finite horizon $T$, the optimal policy at time $t$ depends only on the $$(T-t)$$-hop neighborhood, enabling a Dynamic Programming Principle on the lifted state and justifying Graph Neural Networks as policy/value approximators. It then develops a practical RL blueprint (SMFCRL) that uses a hierarchical meta-policy and local GNN-based policies, with a policy-gradient training recipe and a finite-graph approximation guarantee that $\nabla_\theta J^N(\theta) \to \nabla_\theta J(\theta)$ as $N \to \infty$. Theoretical results show that truncated, shallow GNNs can approximate the optimal policies with exponential accuracy under mild contraction assumptions, and that GNNs possess the necessary expressivity (via Deep Sets and 1-WL) to approximate mean-field objects. Experiments on epidemic-control tasks demonstrate that neighborhood-aware policies recover local heterogeneity and distinguish global structure better than purely mean-field baselines, validating the framework's practical impact for scalable decentralized control on sparse graphs.

Abstract

Mean-field control (MFC) offers a scalable solution to the curse of dimensionality in multi-agent systems but traditionally hinges on the restrictive assumption of exchangeability via dense, all-to-all interactions. In this work, we bridge the gap to real-world network structures by proposing a rigorous framework for MFC on large sparse graphs. We redefine the system state as a probability measure over decorated rooted neighborhoods, effectively capturing local heterogeneity. Our central contribution is a theoretical foundation for scalable reinforcement learning in this setting. We prove horizon-dependent locality: for finite-horizon problems, an agent's optimal policy at time t depends strictly on its (T-t)-hop neighborhood. This result renders the infinite-dimensional control problem tractable and underpins a novel Dynamic Programming Principle (DPP) on the lifted space of neighborhood distributions. Furthermore, we formally and experimentally justify the use of Graph Neural Networks (GNNs) for actor-critic algorithms in this context. Our framework naturally recovers classical MFC as a degenerate case while enabling efficient, theoretically grounded control on complex sparse topologies.

Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions

TL;DR

The paper addresses the curse of dimensionality in multi-agent control on sparse networks by replacing the classical mean-field state with a distribution over decorated rooted neighborhoods, . It proves a horizon-dependent locality theorem: for a finite horizon , the optimal policy at time depends only on the -hop neighborhood, enabling a Dynamic Programming Principle on the lifted state and justifying Graph Neural Networks as policy/value approximators. It then develops a practical RL blueprint (SMFCRL) that uses a hierarchical meta-policy and local GNN-based policies, with a policy-gradient training recipe and a finite-graph approximation guarantee that as . Theoretical results show that truncated, shallow GNNs can approximate the optimal policies with exponential accuracy under mild contraction assumptions, and that GNNs possess the necessary expressivity (via Deep Sets and 1-WL) to approximate mean-field objects. Experiments on epidemic-control tasks demonstrate that neighborhood-aware policies recover local heterogeneity and distinguish global structure better than purely mean-field baselines, validating the framework's practical impact for scalable decentralized control on sparse graphs.

Abstract

Mean-field control (MFC) offers a scalable solution to the curse of dimensionality in multi-agent systems but traditionally hinges on the restrictive assumption of exchangeability via dense, all-to-all interactions. In this work, we bridge the gap to real-world network structures by proposing a rigorous framework for MFC on large sparse graphs. We redefine the system state as a probability measure over decorated rooted neighborhoods, effectively capturing local heterogeneity. Our central contribution is a theoretical foundation for scalable reinforcement learning in this setting. We prove horizon-dependent locality: for finite-horizon problems, an agent's optimal policy at time t depends strictly on its (T-t)-hop neighborhood. This result renders the infinite-dimensional control problem tractable and underpins a novel Dynamic Programming Principle (DPP) on the lifted space of neighborhood distributions. Furthermore, we formally and experimentally justify the use of Graph Neural Networks (GNNs) for actor-critic algorithms in this context. Our framework naturally recovers classical MFC as a degenerate case while enabling efficient, theoretically grounded control on complex sparse topologies.
Paper Structure (28 sections, 12 theorems, 70 equations, 5 figures, 1 algorithm)

This paper contains 28 sections, 12 theorems, 70 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Core Contributions
  4. Background and Preliminaries
  5. From Classical MFC to Graph-Lifted MFC
  6. Sparse Random Graphs and Local Weak Convergence
  7. The Sparse Mean-Field Control Framework
  8. The Finite N-Agent System
  9. The Limiting Sparse-MFC System
  10. Approximation Guarantee
  11. Reinforcement Learning on Sparse Graphs
  12. Horizon-Dependent Local Dynamic Programming
  13. Policy Gradients on Sparse Graphs
  14. Theoretical Analysis of GNN Usage
  15. Approximation by Truncated Policies.
  16. ...and 13 more sections

Key Result

Theorem 1

Let $(G_N)_{N \ge 1}$ be a sequence of graphs converging locally to $(\mathcal{G}, \rho)$. Assume that the space of admissible policies $\Pi$ is compact and that the family $(J^N)_{N\in\mathbb N}$ is equicontinuous (with respect to the topology of weak convergence of policies) on $\Pi$. Then

Figures (5)

  • Figure 1: Overview of the Sparse Mean-Field Control Framework. (Top Left) The physical $N$-agent system on a sparse graph where agents interact locally. (Top Right) The rigorous mean-field limit where the state is redefined as a probability measure $\boldsymbol{\mu}_t$ over the space of decorated rooted graphs. (Bottom) Our Horizon-Dependent Locality result (Theorem \ref{['thm:locality']}) establishes the $(T-t)$-hop neighborhood at time $t$ as sufficient statistic for optimal control.
  • Figure 2: The Sparse Mean-Field RL Architecture. The diagram illustrates the computational flow of Algorithm \ref{['alg:smfac']}. (1) The GNN encodes the local structure of the entire graph $G_N$. (2) A global pooling operation aggregates these embeddings to obtain a function of the mean-field state $\boldsymbol{\mu}_t$. (3) The Meta-Policy (Actor) uses this mean-field state-dependent output to generate the parameters for the local policy $\pi_t$. (Bottom) Individual agents $i$ then execute actions $u_t^i$ by conditioning this shared policy $\pi_t$ on their specific local neighborhoods.
  • Figure 3: The additional information at the local node makes it easier for the agents to decide when to vaccinate, yielding a better policy compared to only local information.
  • Figure 4: Two infection scenarios with identical global counts but different local structure.
  • Figure 5: Applying a GNN encoder on the underlying graph can be essential in finding the optimal policy. Observe that our algorithm is able to find the optimal policy (not vaccinating nodes with infected neighbor if disease is already contained), where as the LWMFMARL is not able to distinguish between the concentrated and scattered initial state.

Theorems & Definitions (25)

  • Definition 1
  • Definition 2: Decorated Rooted Graph
  • Theorem 1: Convergence of Optimal Values
  • proof
  • Corollary 1: Approximate Optimality
  • Theorem 2: Horizon-Dependent Locality
  • Remark 1
  • Theorem 3: Dynamic Programming Principle
  • Theorem 4: Policy Gradient for the Limiting SMFC
  • Theorem 5: Finite-Graph PG Approximation
  • ...and 15 more