Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-free policy gradient for discrete-time mean-field control

Matthieu Meunier, Huyên Pham, Christoph Reisinger

TL;DR

The paper tackles model-free policy gradient learning for discrete-time mean-field control with finite state and compact action spaces, addressing the core challenge that population-dependent transitions and rewards prevent standard likelihood-ratio gradient estimation. It introduces a perturbation scheme on the state-distribution flow, derives a tractable gradient for the perturbed value function, and proves its convergence to the true gradient as the perturbation vanishes. Building on this, the MF-REINFORCE algorithm provides a fully model-free trajectory-based estimator, along with explicit bias and MSE bounds that depend on the perturbation magnitude and sample sizes. Numerical experiments across toy, cybersecurity, and distribution-planning tasks validate the approach and demonstrate practical viability for mean-field policy learning in discrete-time settings.

Abstract

We study model-free policy learning for discrete-time mean-field control (MFC) problems with finite state space and compact action space. In contrast to the extensive literature on value-based methods for MFC, policy-based approaches remain largely unexplored due to the intrinsic dependence of transition kernels and rewards on the evolving population state distribution, which prevents the direct use of likelihood-ratio estimators of policy gradients from classical single-agent reinforcement learning. We introduce a novel perturbation scheme on the state-distribution flow and prove that the gradient of the resulting perturbed value function converges to the true policy gradient as the perturbation magnitude vanishes. This construction yields a fully model-free estimator based solely on simulated trajectories and an auxiliary estimate of the sensitivity of the state distribution. Building on this framework, we develop MF-REINFORCE, a model-free policy gradient algorithm for MFC, and establish explicit quantitative bounds on its bias and mean-squared error. Numerical experiments on representative mean-field control tasks demonstrate the effectiveness of the proposed approach.

Model-free policy gradient for discrete-time mean-field control

TL;DR

The paper tackles model-free policy gradient learning for discrete-time mean-field control with finite state and compact action spaces, addressing the core challenge that population-dependent transitions and rewards prevent standard likelihood-ratio gradient estimation. It introduces a perturbation scheme on the state-distribution flow, derives a tractable gradient for the perturbed value function, and proves its convergence to the true gradient as the perturbation vanishes. Building on this, the MF-REINFORCE algorithm provides a fully model-free trajectory-based estimator, along with explicit bias and MSE bounds that depend on the perturbation magnitude and sample sizes. Numerical experiments across toy, cybersecurity, and distribution-planning tasks validate the approach and demonstrate practical viability for mean-field policy learning in discrete-time settings.

Abstract

We study model-free policy learning for discrete-time mean-field control (MFC) problems with finite state space and compact action space. In contrast to the extensive literature on value-based methods for MFC, policy-based approaches remain largely unexplored due to the intrinsic dependence of transition kernels and rewards on the evolving population state distribution, which prevents the direct use of likelihood-ratio estimators of policy gradients from classical single-agent reinforcement learning. We introduce a novel perturbation scheme on the state-distribution flow and prove that the gradient of the resulting perturbed value function converges to the true policy gradient as the perturbation magnitude vanishes. This construction yields a fully model-free estimator based solely on simulated trajectories and an auxiliary estimate of the sensitivity of the state distribution. Building on this framework, we develop MF-REINFORCE, a model-free policy gradient algorithm for MFC, and establish explicit quantitative bounds on its bias and mean-squared error. Numerical experiments on representative mean-field control tasks demonstrate the effectiveness of the proposed approach.
Paper Structure (27 sections, 12 theorems, 152 equations, 5 figures, 1 table, 2 algorithms)

This paper contains 27 sections, 12 theorems, 152 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Outline of Main Results
  3. Related Works
  4. Policy Gradient Methods in Multi-Agent Problems
  5. Value-Based Methods in the Mean-Field Setting
  6. Perturbation Methods in Reinforcement Learning
  7. Notations and Paper Structure
  8. Policy Gradient Approximation via Perturbation for Mean-Field Reinforcement Learning
  9. Parametrization of State Distributions by Logits
  10. Policy Gradient Formula: from Single Agent to Mean-Field
  11. Perturbed Process and Value Function
  12. Derivation and Convergence of Perturbed Gradient
  13. Policy Gradient Algorithm
  14. Model-Free Estimator
  15. Algorithm
  16. ...and 12 more sections

Key Result

Proposition 2.1

Let Assumption assumption:finite-state-space hold. For any $\mu_0 \in \mathcal{P}(\mathcal{X})^*$ and $\theta \in \Theta$, we have where where $\mathrm{MD(\theta)}, \mathrm{MFD}(\theta)$ stand for Measure Derivative and Mean-Field Derivatve respectively.

Figures (5)

  • Figure 1: Evolution of validation rewards on two-state two-action problem with MF-REINFORCE for different values of $\varepsilon$. The means and standard deviations are computed over 5 independent training runs.
  • Figure 2: Evolution of validation rewards on cybersecurity problem with MF-REINFORCE for different values of $\varepsilon$. The means and standard deviations are computed over 5 independent training runs.
  • Figure 3: Flow of state distribution on cybersecurity problem using the learned policy trained with MF-REINFORCE ($\varepsilon = 1.0$). The dashed lines correspond to the reported state probabilities at the final time step learned via mean-field Q-learning carmona2023model.
  • Figure 4: Evolution of validation rewards on distribution planning problem with MF-REINFORCE for different values of $\varepsilon$. The means and standard deviations are computed over 5 independent training runs.
  • Figure 5: Initial, target, and final distributions using MF-REINFORCE. The final distribution is obtained by generating a trajectory of length $T=5$ after training MF-REINFORCE with different values of $\varepsilon$.

Theorems & Definitions (30)

  • Proposition 2.1
  • Remark 2.1
  • Lemma 2.2
  • Theorem 2.3
  • Remark 2.2
  • Theorem 2.4
  • Remark 2.3
  • Remark 3.1
  • Lemma 3.1
  • Proposition 3.2
  • ...and 20 more