Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Last Iterate Analyses of FTRL in Stochasitc Bandits

Jingxin Zhan, Yuze Han, Zhihua Zhang

TL;DR

This work investigates the last-iterate behavior of Follow-The-Regularized-Leader (FTRL) in stochastic multi-armed bandits, focusing on the $1/2$-Tsallis-INF algorithm with the regularizer $\\Psi(p)=-4\\sum_i\\sqrt{p_i}$. The authors prove that the Bregman divergence $D_{\\Psi}(e_{i_*}, p_t)$ between the optimal-armed point mass and the current arm distribution decays at rate $t^{-1/2}$, establishing the first such last-iterate bound for FTRL in this setting. The analysis introduces a novel decomposition and a continuity property of the iterates, plus a uniform second-moment bound that helps control fluctuations of the estimated regret. Although the $t^{-1/2}$ rate for the Bregman divergence does not immediately imply a $t^{-1}$ last-iterate rate for simple regret, the results provide important insights into the trajectory of FTRL in stochastic bandits and suggest a plausible path toward sharper rates in future work.

Abstract

The convergence analysis of online learning algorithms is central to machine learning theory, where last-iterate convergence is particularly important, as it captures the learner's actual decisions and describes the evolution of the learning process over time. However, in multi-armed bandits, most existing algorithmic analyses mainly focus on the order of regret, while the last-iterate (simple regret) convergence rate remains less explored -- especially for the widely studied Follow-the-Regularized-Leader (FTRL) algorithms. Recently, a growing line of work has established the Best-of-Both-Worlds (BOBW) property of FTRL algorithms in bandit problems, showing in particular that they achieve logarithmic regret in stochastic bandits. Nevertheless, their last-iterate convergence rate has not yet been studied. Intuitively, logarithmic regret should correspond to a $t^{-1}$ last-iterate convergence rate. This paper partially confirms this intuition through theoretical analysis, showing that the Bregman divergence, defined by the regular function $Ψ(p)=-4\sum_{i=1}^{d}\sqrt{p_i}$ associated with the BOBW FTRL algorithm $1/2$-Tsallis-INF (arXiv:1807.07623), between the point mass on the optimal arm and the probability distribution over the arm set obtained at iteration $t$, decays at a rate of $t^{-1/2}$.

Last Iterate Analyses of FTRL in Stochasitc Bandits

TL;DR

This work investigates the last-iterate behavior of Follow-The-Regularized-Leader (FTRL) in stochastic multi-armed bandits, focusing on the -Tsallis-INF algorithm with the regularizer . The authors prove that the Bregman divergence between the optimal-armed point mass and the current arm distribution decays at rate , establishing the first such last-iterate bound for FTRL in this setting. The analysis introduces a novel decomposition and a continuity property of the iterates, plus a uniform second-moment bound that helps control fluctuations of the estimated regret. Although the rate for the Bregman divergence does not immediately imply a last-iterate rate for simple regret, the results provide important insights into the trajectory of FTRL in stochastic bandits and suggest a plausible path toward sharper rates in future work.

Abstract

The convergence analysis of online learning algorithms is central to machine learning theory, where last-iterate convergence is particularly important, as it captures the learner's actual decisions and describes the evolution of the learning process over time. However, in multi-armed bandits, most existing algorithmic analyses mainly focus on the order of regret, while the last-iterate (simple regret) convergence rate remains less explored -- especially for the widely studied Follow-the-Regularized-Leader (FTRL) algorithms. Recently, a growing line of work has established the Best-of-Both-Worlds (BOBW) property of FTRL algorithms in bandit problems, showing in particular that they achieve logarithmic regret in stochastic bandits. Nevertheless, their last-iterate convergence rate has not yet been studied. Intuitively, logarithmic regret should correspond to a last-iterate convergence rate. This paper partially confirms this intuition through theoretical analysis, showing that the Bregman divergence, defined by the regular function associated with the BOBW FTRL algorithm -Tsallis-INF (arXiv:1807.07623), between the point mass on the optimal arm and the probability distribution over the arm set obtained at iteration , decays at a rate of .
Paper Structure (29 sections, 25 theorems, 156 equations, 1 algorithm)

This paper contains 29 sections, 25 theorems, 156 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution
  3. Related Works
  4. Best-of-Both-Worlds (BOBW)
  5. Pure Exploration
  6. Preliminaries
  7. The Problem Setting and Notation
  8. FTRL Policy
  9. Main Results
  10. Last-Iterate Convergence rate
  11. Decomposition
  12. Uniform Bound
  13. Proof of Theorem \ref{['rate']}
  14. Proof Sketch of Lemma \ref{['wedge']}
  15. Proof Sketch of Lemma \ref{['2nd']}
  16. ...and 14 more sections

Key Result

Theorem 3.1

Assume that $i_*$ is unique. For Algorithm alg: FTRL, if $0<\alpha<1$, then for any $t\ge 1$, we have where $C_\alpha=\frac{e^{\alpha^2}}{\alpha^3(1-\alpha^2)}$ and $C$ is a positive constant that is large enough.

Theorems & Definitions (44)

  • Theorem 3.1
  • Corollary 3.2
  • Remark 3.1
  • Remark 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Lemma 3.5
  • Lemma 3.6
  • Lemma 4.1
  • Lemma A.1
  • ...and 34 more