Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Last Iterate Convergence in Monotone Mean Field Games

Noboru Isobe, Kenshi Abe, Kaito Ariu

TL;DR

This paper develops a proximal-point framework with KL regularization to achieve last-iterate convergence for mean-field games under non-strict monotonicity. It proves that PP updates converge to the mean-field equilibrium set and that each PP step is equivalent to solving a KL-regularized MFG, which can be solved exponentially fast via Regularized Mirror Descent. Building on this, the APP algorithm approximates PP with a small number of MD steps, converging to the unregularized equilibrium without time-averaging. Empirical results on the Beach Bar Process validate LIC and show APP reliably reaches the unregularized equilibrium, while exploiting the regularization to ensure robust convergence. This advances LIC in MFGs beyond strictly monotone settings and offers a scalable, algorithmically principled path to mean-field equilibria in practical MARL contexts.

Abstract

In the Lasry--Lions framework, Mean-Field Games (MFGs) model interactions among an infinite number of agents. However, existing algorithms either require strict monotonicity or only guarantee the convergence of averaged iterates, as in Fictitious Play in continuous time. We address this gap with the following theoretical result. First, we prove that the last-iterated policy of a proximal-point (PP) update with KL regularization converges to an equilibrium of MFG under non-strict monotonicity. Second, we see that each PP update is equivalent to finding the equilibria of a KL-regularized MFG. We then prove that this equilibrium can be found using Mirror Descent (MD) with an exponential last-iterate convergence rate. Building on these insights, we propose the Approximate Proximal-Point ($\mathtt{APP}$) algorithm, which approximately implements the PP update via a small number of MD steps. Numerical experiments on standard benchmarks confirm that the $\mathtt{APP}$ algorithm reliably converges to the unregularized mean-field equilibrium without time-averaging.

Last Iterate Convergence in Monotone Mean Field Games

TL;DR

This paper develops a proximal-point framework with KL regularization to achieve last-iterate convergence for mean-field games under non-strict monotonicity. It proves that PP updates converge to the mean-field equilibrium set and that each PP step is equivalent to solving a KL-regularized MFG, which can be solved exponentially fast via Regularized Mirror Descent. Building on this, the APP algorithm approximates PP with a small number of MD steps, converging to the unregularized equilibrium without time-averaging. Empirical results on the Beach Bar Process validate LIC and show APP reliably reaches the unregularized equilibrium, while exploiting the regularization to ensure robust convergence. This advances LIC in MFGs beyond strictly monotone settings and offers a scalable, algorithmically principled path to mean-field equilibria in practical MARL contexts.

Abstract

In the Lasry--Lions framework, Mean-Field Games (MFGs) model interactions among an infinite number of agents. However, existing algorithms either require strict monotonicity or only guarantee the convergence of averaged iterates, as in Fictitious Play in continuous time. We address this gap with the following theoretical result. First, we prove that the last-iterated policy of a proximal-point (PP) update with KL regularization converges to an equilibrium of MFG under non-strict monotonicity. Second, we see that each PP update is equivalent to finding the equilibria of a KL-regularized MFG. We then prove that this equilibrium can be found using Mirror Descent (MD) with an exponential last-iterate convergence rate. Building on these insights, we propose the Approximate Proximal-Point () algorithm, which approximately implements the PP update via a small number of MD steps. Numerical experiments on standard benchmarks confirm that the algorithm reliably converges to the unregularized mean-field equilibrium without time-averaging.
Paper Structure (37 sections, 16 theorems, 80 equations, 3 figures, 1 table, 2 algorithms)

This paper contains 37 sections, 16 theorems, 80 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem setting and preliminary facts
  3. Notation:
  4. Mean-field games
  5. Proximal point-type method for MFG
  6. Last-iterate convergence result
  7. Approximating proximal point with mirror descent in regularized MFG
  8. Approximation of the update rule of PP with regularized MFG
  9. An exponential convergence result
  10. Intuition for exponential convergence: continuous-time version of Regularized Mirror Descent
  11. Proof sketch of the convergence result for regularized mirror descent
  12. APP: approximating PP updates with RMD
  13. Numerical experiment
  14. Limitations
  15. Related works
  16. ...and 22 more sections

Key Result

Theorem 3.1

Let $(\sigma^k)_{k=0}^\infty$ be the sequence defined by eq:slingshot_update. In addition to assump:transition_kernelassump:Lipassump:monotonicity, assume that the initial policy $\pi^0$ has full support, i.e., $\min_{(h,s,a)\in [H]\times\mathcal{S}\times\mathcal{A}}\pi^0_h \left(a \;\middle\vert\;

Figures (3)

  • Figure 1: Behavior of $\mathop{\mathrm{\mathtt{RMD}}}\nolimits$.
  • Figure 2: Experimental results for \ref{['alg:reward_transform']} for Beach Bar Process
  • Figure : $\mathop{\mathrm{\mathtt{APP}}}\nolimits$ for MFG

Theorems & Definitions (24)

  • Remark 2.2
  • Definition 2.3
  • Theorem 3.1
  • Lemma 3.2
  • Remark 3.3: Challenges in the proof of \ref{['thm:last_iterate_conv']}
  • Definition 4.2
  • Theorem 4.3
  • Remark 4.4
  • Theorem 4.5
  • Claim 4.6
  • ...and 14 more