Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of Time-Averaged Mean Field Gradient Descent Dynamics for Continuous Multi-Player Zero-Sum Games

Yulong Lu, Pierre Monmarché

TL;DR

This work extends mean-field gradient descent analysis to multi-player zero-sum games ($K\ge 2$) by introducing a time-averaged, momentum-enhanced dynamics on the distributions of mixed strategies with entropy regularization. For fixed temperature $\tau$, the authors prove exponential convergence to the unique MNE of the regularized energies ${\mathcal E}_{i,\tau}$, with an explicit rate depending on $\alpha,\tau$ and game bounds. They further show that an annealed schedule with $\tau_t\to 0$ yields convergence to an MNE of the original unregularized game in the Nikaido-Isoda error, at a rate $O(\log\log t/\log t)$. The results leverage entropy-entropy production arguments and exploit the zero-sum structure to control cross-player couplings without relying on scale separation, enabling uniform treatment of all players on the same time scale. The theoretical findings pave the way for practical, scalable mean-field algorithms for computing MNEs in complex multi-player settings with potential applications to adversarial ML and multi-agent reinforcement learning.

Abstract

The approximation of mixed Nash equilibria (MNE) for zero-sum games with mean-field interacting players has recently raised much interest in machine learning. In this paper we propose a mean-field gradient descent dynamics for finding the MNE of zero-sum games involving $K$ players with $K\geq 2$. The evolution of the players' strategy distributions follows coupled mean-field gradient descent flows with momentum, incorporating an exponentially discounted time-averaging of gradients. First, in the case of a fixed entropic regularization, we prove an exponential convergence rate for the mean-field dynamics to the mixed Nash equilibrium with respect to the total variation metric. This improves a previous polynomial convergence rate for a similar time-averaged dynamics with different averaging factors. Moreover, unlike previous two-scale approaches for finding the MNE, our approach treats all player types on the same time scale. We also show that with a suitable choice of decreasing temperature, a simulated annealing version of the mean-field dynamics converges to an MNE of the initial unregularized problem.

Convergence of Time-Averaged Mean Field Gradient Descent Dynamics for Continuous Multi-Player Zero-Sum Games

TL;DR

This work extends mean-field gradient descent analysis to multi-player zero-sum games () by introducing a time-averaged, momentum-enhanced dynamics on the distributions of mixed strategies with entropy regularization. For fixed temperature , the authors prove exponential convergence to the unique MNE of the regularized energies , with an explicit rate depending on and game bounds. They further show that an annealed schedule with yields convergence to an MNE of the original unregularized game in the Nikaido-Isoda error, at a rate . The results leverage entropy-entropy production arguments and exploit the zero-sum structure to control cross-player couplings without relying on scale separation, enabling uniform treatment of all players on the same time scale. The theoretical findings pave the way for practical, scalable mean-field algorithms for computing MNEs in complex multi-player settings with potential applications to adversarial ML and multi-agent reinforcement learning.

Abstract

The approximation of mixed Nash equilibria (MNE) for zero-sum games with mean-field interacting players has recently raised much interest in machine learning. In this paper we propose a mean-field gradient descent dynamics for finding the MNE of zero-sum games involving players with . The evolution of the players' strategy distributions follows coupled mean-field gradient descent flows with momentum, incorporating an exponentially discounted time-averaging of gradients. First, in the case of a fixed entropic regularization, we prove an exponential convergence rate for the mean-field dynamics to the mixed Nash equilibrium with respect to the total variation metric. This improves a previous polynomial convergence rate for a similar time-averaged dynamics with different averaging factors. Moreover, unlike previous two-scale approaches for finding the MNE, our approach treats all player types on the same time scale. We also show that with a suitable choice of decreasing temperature, a simulated annealing version of the mean-field dynamics converges to an MNE of the initial unregularized problem.
Paper Structure (18 sections, 7 theorems, 89 equations)

This paper contains 18 sections, 7 theorems, 89 equations.

Table of Contents

  1. Introduction
  2. Two-player zero-sum games
  3. Time-averaged mean-field gradient descent dynamics.
  4. Summary of contributions
  5. Related works
  6. Computational complexity of zero-sum polymatrix games
  7. Continuous two-player games
  8. Mean field analysis
  9. General notations
  10. Assumptions and Main Results
  11. Settings and notations
  12. Convergence time-averaged mean-field gradient descent dynamics at a fixed temperature
  13. Convergence of annealed dynamics
  14. Proofs of Main Results
  15. Proof of the result in Section \ref{['sec:setting']}
  16. ...and 3 more sections

Key Result

Proposition 2.2

Under Assumption ass-multi, for any $\tau>0$, there exists a unique Nash equilibrium $\nu^\ast_\tau = (\nu^{1,\ast}_{\tau},\dots,\nu^{K,\ast}_{\tau})$ associated to the energies $\{{\mathcal{E}}_{i,\tau}\}_{i=1}^K$. Moreover, for any probability measures $\nu = (\nu^1,\dots,\nu^K)$, it holds that

Theorems & Definitions (16)

  • Proposition 2.2
  • Proposition 2.3
  • Theorem 2.4
  • Remark 2.5
  • Proposition 2.6
  • Theorem 2.7
  • Theorem 2.8
  • Remark 2.9
  • Proof 1: Proof of Proposition \ref{['prop:NEmulti']}
  • Proof 2: Proof of Proposition \ref{['prop:Lyapunov-multi']}
  • ...and 6 more