Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds
Taira Tsuchiya, Shinji Ito, Junya Honda
TL;DR
This work introduces the stability-penalty-adaptive (SPA) learning rate for Follow-the-regularized-leader, unifying stability and regularization penalty to bound regret in adversarial and stochastic sequential decision problems. By carefully decomposing regret into stability, penalty, and transformation terms, SPA yields three key adaptivity axes: sparsity, game-dependency, and best-of-both-worlds (BOBW) guarantees, across multi-armed bandits and partial-monitoring scenarios. The authors develop s-agnostic sparsity-dependent bounds, a sparsity-aware BOBW algorithm, and a PM game-dependent bound, employing time-varying learning rates, the time-invariant log-barrier, and an advanced variation analysis of the FTRL output under changing regularizers. The results advance practical adaptability in online learning by achieving near-optimal sparsity- and game-dependent performance, as well as robust BOBW guarantees in both adversarial and stochastic regimes.
Abstract
Adaptivity to the difficulties of a problem is a key property in sequential decision-making problems to broaden the applicability of algorithms. Follow-the-regularized-leader (FTRL) has recently emerged as one of the most promising approaches for obtaining various types of adaptivity in bandit problems. Aiming to further generalize this adaptivity, we develop a generic adaptive learning rate, called stability-penalty-adaptive (SPA) learning rate for FTRL. This learning rate yields a regret bound jointly depending on stability and penalty of the algorithm, into which the regret of FTRL is typically decomposed. With this result, we establish several algorithms with three types of adaptivity: sparsity, game-dependency, and best-of-both-worlds (BOBW). Despite the fact that sparsity appears frequently in real problems, existing sparse multi-armed bandit algorithms with $k$-arms assume that the sparsity level $s \leq k$ is known in advance, which is often not the case in real-world scenarios. To address this issue, we first establish $s$-agnostic algorithms with regret bounds of $\tilde{O}(\sqrt{sT})$ in the adversarial regime for $T$ rounds, which matches the existing lower bound up to a logarithmic factor. Meanwhile, BOBW algorithms aim to achieve a near-optimal regret in both the stochastic and adversarial regimes. Leveraging the SPA learning rate and the technique for $s$-agnostic algorithms combined with a new analysis to bound the variation in FTRL output in response to changes in a regularizer, we establish the first BOBW algorithm with a sparsity-dependent bound. Additionally, we explore partial monitoring and demonstrate that the proposed SPA learning rate framework allows us to achieve a game-dependent bound and the BOBW simultaneously.