Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Last-iterate Convergence Separation between Extra-gradient and Optimism in Constrained Periodic Games

Yi Feng, Ping Li, Ioannis Panageas, Xiao Wang

TL;DR

The work investigates last-iterate convergence of optimistic and extra-gradient multiplicative weights updates in constrained, time-varying (periodic) zero-sum games. It demonstrates a sharp separation: there exists a constrained 2-periodic game with a common equilibrium where OMWU fails to converge (and drifts to the boundary), while Extra-MWU converges to the equilibrium when the periodic game has a common fully mixed equilibrium and the step size satisfies $\eta \max_t \|A_t\|<1$. The results generalize prior unconstrained findings to practical constrained settings, leveraging KL-divergence as a Lyapunov-like measure and discrete-time LaSalle arguments for convergence. Numerical experiments corroborate the theory, and the paper discusses dynamics in games without a common equilibrium, where Extra-MWU ends up on a $\mathcal{T}$-periodic orbit and OMWU diverges.

Abstract

Last-iterate behaviors of learning algorithms in repeated two-player zero-sum games have been extensively studied due to their wide applications in machine learning and related tasks. Typical algorithms that exhibit the last-iterate convergence property include optimistic and extra-gradient methods. However, most existing results establish these properties under the assumption that the game is time-independent. Recently, (Feng et al, 2023) studied the last-iterate behaviors of optimistic and extra-gradient methods in games with a time-varying payoff matrix, and proved that in an unconstrained periodic game, extra-gradient method converges to the equilibrium while optimistic method diverges. This finding challenges the conventional wisdom that these two methods are expected to behave similarly as they do in time-independent games. However, compared to unconstrained games, games with constrains are more common both in practical and theoretical studies. In this paper, we investigate the last-iterate behaviors of optimistic and extra-gradient methods in the constrained periodic games, demonstrating that similar separation results for last-iterate convergence also hold in this setting.

Last-iterate Convergence Separation between Extra-gradient and Optimism in Constrained Periodic Games

TL;DR

The work investigates last-iterate convergence of optimistic and extra-gradient multiplicative weights updates in constrained, time-varying (periodic) zero-sum games. It demonstrates a sharp separation: there exists a constrained 2-periodic game with a common equilibrium where OMWU fails to converge (and drifts to the boundary), while Extra-MWU converges to the equilibrium when the periodic game has a common fully mixed equilibrium and the step size satisfies . The results generalize prior unconstrained findings to practical constrained settings, leveraging KL-divergence as a Lyapunov-like measure and discrete-time LaSalle arguments for convergence. Numerical experiments corroborate the theory, and the paper discusses dynamics in games without a common equilibrium, where Extra-MWU ends up on a -periodic orbit and OMWU diverges.

Abstract

Last-iterate behaviors of learning algorithms in repeated two-player zero-sum games have been extensively studied due to their wide applications in machine learning and related tasks. Typical algorithms that exhibit the last-iterate convergence property include optimistic and extra-gradient methods. However, most existing results establish these properties under the assumption that the game is time-independent. Recently, (Feng et al, 2023) studied the last-iterate behaviors of optimistic and extra-gradient methods in games with a time-varying payoff matrix, and proved that in an unconstrained periodic game, extra-gradient method converges to the equilibrium while optimistic method diverges. This finding challenges the conventional wisdom that these two methods are expected to behave similarly as they do in time-independent games. However, compared to unconstrained games, games with constrains are more common both in practical and theoretical studies. In this paper, we investigate the last-iterate behaviors of optimistic and extra-gradient methods in the constrained periodic games, demonstrating that similar separation results for last-iterate convergence also hold in this setting.
Paper Structure (25 sections, 32 theorems, 174 equations, 9 figures)

This paper contains 25 sections, 32 theorems, 174 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Our contribution.
  3. Technical Comparison.
  4. Organization.
  5. Preliminaries
  6. Game theory
  7. Periodic zero-sum game.
  8. Learning dynamics in games
  9. MWU.
  10. Optimistic MWU.
  11. Extra-gradient MWU.
  12. Results from dynamical systems
  13. Main Results
  14. Outline of the Proof
  15. Proofs of Theorem \ref{['thm: OMWU fails']}
  16. ...and 10 more sections

Key Result

Proposition 2.3

Let $\Tilde{f}_i = f_{i+{\mathcal{T}}-1} \circ ... \circ f_{i}$, for $i \in [{\mathcal{T}}]$. Then $\Tilde{f}_i$ is a time-independent dynamical system. If for all $x \in {\mathcal{X}}$ and each $i\in [{\mathcal{T}}]$, it holds that $\lim_{n \to \infty} \Tilde{f}_i^n(x) = x^*$ for some $x^* \in {\ma

Figures (9)

  • Figure 1: KL-divergence of OMWU in periodic game.
  • Figure 2: Trajectories of strategies for a player when using Extra-MWU in the periodic game defined in (\ref{['2-periodic game_m']}).
  • Figure 3: Curves composed of the fixed points of ${\mathcal{G}}_1 \circ {\mathcal{G}}_2$.
  • Figure 4: First experimental results for Extra-MWU.
  • Figure 5: First experimental results for OMWU.
  • ...and 4 more figures

Theorems & Definitions (58)

  • Definition 2.1: Periodic zero-sum games
  • Definition 2.2: Periodic dynamical system
  • Proposition 2.3: franke2003attractors
  • Definition 2.4: Stable, Unstable, and Center eigenspaces.
  • Proposition 2.5: galor2007discrete
  • Theorem 3.1
  • Theorem 3.2
  • Proposition 4.0
  • Proposition 4.0
  • Proposition 4.0
  • ...and 48 more