Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Perturbed Gradient Ascent for Last-Iterate Convergence in Games

Kenshi Abe, Mitsuki Sakamoto, Kaito Ariu, Atsushi Iwasaki

TL;DR

Gradient Ascent with Boosting Payoff Perturbation is proposed, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme.

Abstract

This paper presents a payoff perturbation technique, introducing a strong convexity to players' payoff functions in games. This technique is specifically designed for first-order methods to achieve last-iterate convergence in games where the gradient of the payoff functions is monotone in the strategy profile space, potentially containing additive noise. Although perturbation is known to facilitate the convergence of learning algorithms, the magnitude of perturbation requires careful adjustment to ensure last-iterate convergence. Previous studies have proposed a scheme in which the magnitude is determined by the distance from a periodically re-initialized anchoring or reference strategy. Building upon this, we propose Gradient Ascent with Boosting Payoff Perturbation, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme. This innovation empowers us to provide faster last-iterate convergence rates against the existing payoff perturbed algorithms, even in the presence of additive noise.

Boosting Perturbed Gradient Ascent for Last-Iterate Convergence in Games

TL;DR

Gradient Ascent with Boosting Payoff Perturbation is proposed, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme.

Abstract

This paper presents a payoff perturbation technique, introducing a strong convexity to players' payoff functions in games. This technique is specifically designed for first-order methods to achieve last-iterate convergence in games where the gradient of the payoff functions is monotone in the strategy profile space, potentially containing additive noise. Although perturbation is known to facilitate the convergence of learning algorithms, the magnitude of perturbation requires careful adjustment to ensure last-iterate convergence. Previous studies have proposed a scheme in which the magnitude is determined by the distance from a periodically re-initialized anchoring or reference strategy. Building upon this, we propose Gradient Ascent with Boosting Payoff Perturbation, which incorporates a novel perturbation into the underlying payoff function, maintaining the periodically re-initializing anchoring strategy scheme. This innovation empowers us to provide faster last-iterate convergence rates against the existing payoff perturbed algorithms, even in the presence of additive noise.
Paper Structure (37 sections, 10 theorems, 114 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 37 sections, 10 theorems, 114 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Monotone games.
  4. Nash equilibrium and gap function.
  5. Problem setting.
  6. Payoff-perturbed learning algorithms.
  7. Gradient ascent with boosting payoff perturbation
  8. Last-iterate convergence rates
  9. Full feedback setting
  10. Noisy feedback setting
  11. Proof sketch of Theorems \ref{['thm:lic_rate_in_full_fb']} and \ref{['thm:lic_rate_in_noisy_fb']}
  12. (1) Potential function for bounding the distance between $\pi^{\mu, k(t)}$ and $\sigma^{k(t)}$.
  13. (2) Convergence rate of $\sigma^{k(t)+1}$ to the stationary point $\pi^{\mu, k(t)}$.
  14. (3) Decomposition of the gap function of the last-iterate strategy profile $\pi^{T+1}$.
  15. (4) Putting it all together: last-iterate convergence rate of $\pi^{T+1}$.
  16. ...and 22 more sections

Key Result

Lemma 2.3

For any $\pi \in \mathcal{X}$, we have:

Figures (3)

  • Figure 1: The gap function for $\pi^t$ of GABP, APGA, OG, and AOG with full and noisy feedback. The shaded area represents the standard errors.
  • Figure 2: Dynamic regret for GABP, APGA, OG, and AOG with full and noisy feedback.
  • Figure : GABP for player $i$.

Theorems & Definitions (20)

  • Example 2.1: Concave-convex games
  • Example 2.2: Cournot Competition
  • Lemma 2.3: Lemma 2 of cai2022finite
  • Theorem 4.1
  • Theorem 4.3
  • Theorem 5.1
  • proof : Proof of Theorem \ref{['thm:lic_rate_in_full_fb']}
  • Lemma B.1
  • Lemma B.2
  • proof : Proof of Lemma \ref{['lem:convergence_rate_of_inner_loop_in_full_fb']}
  • ...and 10 more