Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

René Carmona, Mathieu Laurière, Zongjun Tan

TL;DR

The paper addresses learning optimal policies for cooperative, large-scale mean-field MDPs with common noise, by proving global convergence guarantees for policy gradient methods in a Linear-Quadratic MF setting. It shows that the MF optimal policy is linear in the state and conditional mean, derivable via a discrete-time Pontryagin principle and Riccati equations, and provides exact and model-free PG algorithms with rigorous convergence rates. The work introduces two practical model-free approaches—a MKV-based zeroth-order estimator and a population-based estimator—and derives finite-sample convergence bounds, along with a detailed perturbation analysis to handle noise and finite-N effects. The results have practical implications for scalable, centralized or cooperative learning in very large populations, with robust performance demonstrated through numerical experiments under varying population sizes and discount factors.

Abstract

We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

TL;DR

The paper addresses learning optimal policies for cooperative, large-scale mean-field MDPs with common noise, by proving global convergence guarantees for policy gradient methods in a Linear-Quadratic MF setting. It shows that the MF optimal policy is linear in the state and conditional mean, derivable via a discrete-time Pontryagin principle and Riccati equations, and provides exact and model-free PG algorithms with rigorous convergence rates. The work introduces two practical model-free approaches—a MKV-based zeroth-order estimator and a population-based estimator—and derives finite-sample convergence bounds, along with a detailed perturbation analysis to handle noise and finite-N effects. The results have practical implications for scalable, centralized or cooperative learning in very large populations, with robust performance demonstrated through numerical experiments under varying population sizes and discount factors.

Abstract

We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.

Paper Structure

This paper contains 46 sections, 44 theorems, 293 equations, 8 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Models
  3. Model for a finite number of agents
  4. The limit mean-field control model
  5. Standing assumptions and well-definiteness
  6. Analysis of the MFC Problem
  7. Admissibility of the optimal control
  8. Duality and Pontryagin Maximum Principle
  9. Identification of the optimal control
  10. Proof of Theorem \ref{['thm:existence_linear_control']}
  11. Policy gradient algorithms
  12. Reformulation of the MFC problem
  13. Additional notations
  14. Exact PG for MFC
  15. Model free PG for MFC with an MKV simulator
  16. ...and 31 more sections

Key Result

Lemma 1

If $\gamma \| {\mathrm A} \|^2 < 1$, if $(\beta_t)_{t \geq 0} \in {\mathcal{U}}_{ad}$, and if $Y_0$ is ${\mathcal{F}}_0$-measurable and square-integrable, a process ${\mathbf Y} = (Y_t)_{t \geq 0}$ satisfying belongs to ${\mathcal{X}}$.

Figures (8)

  • Figure 1: Comparison of costs achieved by the controls learned with MKV and population simulator using model-free PG. (a) : MF costs; (b) : relative error in MF cost w.r.t. MF optimum; (c) : $N$-agent costs and their respective optimal value; (d) : relative error in $N$-agent costs w.r.t. $N$-agent optimal cost.
  • Figure 2: Convergence of the control parameters. (a) $1$-agent simulator. (b) MKV and population simulators.
  • Figure 3: Comparison of the $N$ agent optimal control and the MKV optimal control. (a): Maximum difference between diagonal terms; (b) Social cost for each control.
  • Figure 4: Influence of the heterogeneity degree on the social cost.
  • Figure 5: Discount factor $\gamma = 0.5$, heterogeneous perturbation size $\tilde{h}=0.4$.
  • ...and 3 more figures

Theorems & Definitions (96)

  • Lemma 1
  • proof
  • Proposition 2
  • proof
  • Corollary 3
  • proof
  • Remark 4
  • Definition 5
  • Lemma 6
  • proof
  • ...and 86 more