Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instance-Dependent Regret Bounds for Learning Two-Player Zero-Sum Games with Bandit Feedback

Shinji Ito, Haipeng Luo, Taira Tsuchiya, Yue Wu

TL;DR

This work extends no-regret self-play learning to the bandit-feedback setting in two-player zero-sum normal-form games by applying Tsallis-INF to both players. It derives instance-dependent regret bounds of the form $O(c_1 \log T + \sqrt{c_2 T})$, where $c_1$ and $c_2$ capture the difficulty of learning via the NE supports and gap structure; notably, $c_2$ vanishes when a PSNE exists, yielding an optimal rate. The results further establish last-iterate convergence under PSNE and show that the PSNE can be identified with near-optimal sample complexity, all within a decentralized, uncoupled learning framework. Empirical experiments corroborate the theory, showing improved regret behavior and PSNE identification in large-scale settings, and highlighting the practical relevance for learning in uncertain strategic environments.

Abstract

No-regret self-play learning dynamics have become one of the premier ways to solve large-scale games in practice. Accelerating their convergence via improving the regret of the players over the naive $O(\sqrt{T})$ bound after $T$ rounds has been extensively studied in recent years, but almost all studies assume access to exact gradient feedback. We address the question of whether acceleration is possible under bandit feedback only and provide an affirmative answer for two-player zero-sum normal-form games. Specifically, we show that if both players apply the Tsallis-INF algorithm of Zimmert and Seldin (2018, arXiv:1807.07623), then their regret is at most $O(c_1 \log T + \sqrt{c_2 T})$, where $c_1$ and $c_2$ are game-dependent constants that characterize the difficulty of learning -- $c_1$ resembles the complexity of learning a stochastic multi-armed bandit instance and depends inversely on some gap measures, while $c_2$ can be much smaller than the number of actions when the Nash equilibria have a small support or are close to the boundary. In particular, for the case when a pure strategy Nash equilibrium exists, $c_2$ becomes zero, leading to an optimal instance-dependent regret bound as we show. We additionally prove that in this case, our algorithm also enjoys last-iterate convergence and can identify the pure strategy Nash equilibrium with near-optimal sample complexity.

Instance-Dependent Regret Bounds for Learning Two-Player Zero-Sum Games with Bandit Feedback

TL;DR

This work extends no-regret self-play learning to the bandit-feedback setting in two-player zero-sum normal-form games by applying Tsallis-INF to both players. It derives instance-dependent regret bounds of the form , where and capture the difficulty of learning via the NE supports and gap structure; notably, vanishes when a PSNE exists, yielding an optimal rate. The results further establish last-iterate convergence under PSNE and show that the PSNE can be identified with near-optimal sample complexity, all within a decentralized, uncoupled learning framework. Empirical experiments corroborate the theory, showing improved regret behavior and PSNE identification in large-scale settings, and highlighting the practical relevance for learning in uncertain strategic environments.

Abstract

No-regret self-play learning dynamics have become one of the premier ways to solve large-scale games in practice. Accelerating their convergence via improving the regret of the players over the naive bound after rounds has been extensively studied in recent years, but almost all studies assume access to exact gradient feedback. We address the question of whether acceleration is possible under bandit feedback only and provide an affirmative answer for two-player zero-sum normal-form games. Specifically, we show that if both players apply the Tsallis-INF algorithm of Zimmert and Seldin (2018, arXiv:1807.07623), then their regret is at most , where and are game-dependent constants that characterize the difficulty of learning -- resembles the complexity of learning a stochastic multi-armed bandit instance and depends inversely on some gap measures, while can be much smaller than the number of actions when the Nash equilibria have a small support or are close to the boundary. In particular, for the case when a pure strategy Nash equilibrium exists, becomes zero, leading to an optimal instance-dependent regret bound as we show. We additionally prove that in this case, our algorithm also enjoys last-iterate convergence and can identify the pure strategy Nash equilibrium with near-optimal sample complexity.

Paper Structure

This paper contains 19 sections, 16 theorems, 108 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Notations
  5. Two-Player Zero-Sum Normal-Form Games
  6. Learning Dynamics with Bandit Feedback
  7. Tsallis-INF Algorithm
  8. Instance-Dependent Regret Bounds for General Zero-Sum Games
  9. Games with Pure-Strategy Nash Equilibria
  10. Regret lower bound
  11. Last-iterate convergence
  12. Sample complexity of identifying PSNE
  13. Numerical Experiments
  14. Conclusions
  15. Technical Lemmas
  16. ...and 4 more sections

Key Result

Theorem 1

For any $x \in \mathcal{P}_m$, the pseudo-regret of the Tsallis-INF algorithm against $x$ is bounded as follows for the row player (and similarly for the column players): where $C_1$ and $C_2$ are positive universal constants and $D(x', x) = \sum_{i=1}^m \frac{1}{\sqrt{x(i)}}(\sqrt{x'(i)} - \sqrt{x(i)})^2$ is the Bregman divergence associated with the $\frac{1}{2}$-Tsallis entropy. In particular,

Figures (2)

  • Figure 1: Regret scaling for Tsallis-INF and two other bandit algorithms. Each configuration $(T)$ is run for 512 trials. The interval between the 10th and 90th percentile is overlaid. The thicker dashed line represents a linear fit on the $T\geq 10^5$ subset of the log-log data.
  • Figure 2: Experimental validation of Tsallis-INF's PSNE identification capability. The plot shows the algorithm's success rate in correctly identifying PSNE against the number of itrations. We use a hard instance of a $256\times 256$ matrix and $\Delta_1=0.1$, running 512 trials for each $\Delta_{\min}$ values over a horizon of $128\mathrm{OPT}$ iterations, where $\mathrm{OPT}$ is the theoretical lower bound for PSNE identification. The $x$-axis is scaled by $1/\mathrm{OPT}$.

Theorems & Definitions (33)

  • Theorem 1: zimmert2021tsallis
  • Theorem 2
  • Corollary 1
  • Theorem 3
  • Remark 1
  • Remark 2
  • Proposition 1
  • proof
  • Theorem 4
  • proof
  • ...and 23 more