Doubly Optimal No-Regret Online Learning in Strongly Monotone Games with Bandit Feedback
Wenjia Ba, Tianyi Lin, Jiawei Zhang, Zhengyuan Zhou
TL;DR
This work tackles online no-regret learning in unknown multi-agent games under bandit feedback, where only individual rewards are observed. It introduces a barrier-based bandit algorithm that combines self-concordant barrier regularization, a single-shot ellipsoidal gradient estimator, and mirror descent to achieve the minimax regret bound tilde{Θ}(n10sqrt{T}) in the single-agent setting. Extending to N-player strongly monotone games, the authors prove that if all players use the same algorithm, the joint action converges in the last iterate to the unique Nash equilibrium at rate tilde{Θ}(nT^{-1/2}). This yields a doubly optimal bandit learner (up to log factors), matching the best possible regret in single-agent learning and the best known last-iterate convergence rate in multi-agent learning. Numerical experiments on Cournot competition and Kelly auctions confirm improved iteration counts over prior multi-agent bandit methods, supporting the theoretical gains and practical viability of the approach.
Abstract
We consider online no-regret learning in unknown games with bandit feedback, where each player can only observe its reward at each time -- determined by all players' current joint action -- rather than its gradient. We focus on the class of \textit{smooth and strongly monotone} games and study optimal no-regret learning therein. Leveraging self-concordant barrier functions, we first construct a new bandit learning algorithm and show that it achieves the single-agent optimal regret of $\tildeΘ(n\sqrt{T})$ under smooth and strongly concave reward functions ($n \geq 1$ is the problem dimension). We then show that if each player applies this no-regret learning algorithm in strongly monotone games, the joint action converges in the \textit{last iterate} to the unique Nash equilibrium at a rate of $\tildeΘ(nT^{-1/2})$. Prior to our work, the best-known convergence rate in the same class of games is $\tilde{O}(n^{2/3}T^{-1/3})$ (achieved by a different algorithm), thus leaving open the problem of optimal no-regret learning algorithms (since the known lower bound is $Ω(nT^{-1/2})$). Our results thus settle this open problem and contribute to the broad landscape of bandit game-theoretical learning by identifying the first doubly optimal bandit learning algorithm, in that it achieves (up to log factors) both optimal regret in the single-agent learning and optimal last-iterate convergence rate in the multi-agent learning. We also present preliminary numerical results on several application problems to demonstrate the efficacy of our algorithm in terms of iteration count.