Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization
Constantinos Daskalakis, Ioannis Panageas
TL;DR
This paper studies constrained two-player zero-sum min–max problems on the simplices $\Delta_n\times\Delta_m$ with payoff $\mathbf{x}^\top A\mathbf{y}$ and proves that Optimistic Multiplicative-Weights Update (OMWU) achieves last-iterate convergence to the unique minimax solution under a small constant stepsize $\eta$, starting from the uniform distribution. The authors use a dynamical-systems framework to show that the Kullback–Leibler divergence to the optimum decreases by at least $\Omega(\eta^3)$ per step until iterates enter a neighborhood, after which a Jacobian-based local contraction ensures convergence of the last iterate. The results extend prior unconstrained last-iterate convergence results to constrained settings and introduce two-phase analysis that may be applicable to other learning dynamics in game-theoretic and optimization contexts. This has implications for practical training regimes in problems like GANs, where constrained min–max dynamics are prevalent and stable, pointwise convergence is desirable.
Abstract
Motivated by applications in Game Theory, Optimization, and Generative Adversarial Networks, recent work of Daskalakis et al \cite{DISZ17} and follow-up work of Liang and Stokes \cite{LiangS18} have established that a variant of the widely used Gradient Descent/Ascent procedure, called "Optimistic Gradient Descent/Ascent (OGDA)", exhibits last-iterate convergence to saddle points in {\em unconstrained} convex-concave min-max optimization problems. We show that the same holds true in the more general problem of {\em constrained} min-max optimization under a variant of the no-regret Multiplicative-Weights-Update method called "Optimistic Multiplicative-Weights Update (OMWU)". This answers an open question of Syrgkanis et al \cite{SALS15}. The proof of our result requires fundamentally different techniques from those that exist in no-regret learning literature and the aforementioned papers. We show that OMWU monotonically improves the Kullback-Leibler divergence of the current iterate to the (appropriately normalized) min-max solution until it enters a neighborhood of the solution. Inside that neighborhood we show that OMWU is locally (asymptotically) stable converging to the exact solution. We believe that our techniques will be useful in the analysis of the last iterate of other learning algorithms.