Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Gradient Succeeds for Bandits

Jincheng Mei, Zixin Zhong, Bo Dai, Alekh Agarwal, Csaba Szepesvari, Dale Schuurmans

TL;DR

This paper proves that the canonical stochastic gradient bandit algorithm achieves almost-sure global convergence to a globally optimal policy at an $O(1/t)$ rate with a constant learning rate. The key innovations are a strong growth condition that makes the stochastic gradient noise naturally decay in tandem with progress, and an automatic weak exploration mechanism via the softmax Jacobian that prevents probabilities from collapsing too quickly. The authors develop non-uniform smoothness and Łojasiewicz-type tools to bound noise and establish both asymptotic convergence and finite-time rates, with experimental results corroborating the theory. The findings imply that simple gradient-based bandits can balance exploration and exploitation without decaying steps, offering practical benefits and potential extensions to broader reinforcement learning settings.

Abstract

We show that the \emph{stochastic gradient} bandit algorithm converges to a \emph{globally optimal} policy at an $O(1/t)$ rate, even with a \emph{constant} step size. Remarkably, global convergence of the stochastic gradient bandit algorithm has not been previously established, even though it is an old algorithm known to be applicable to bandits. The new result is achieved by establishing two novel technical findings: first, the noise of the stochastic updates in the gradient bandit algorithm satisfies a strong ``growth condition'' property, where the variance diminishes whenever progress becomes small, implying that additional noise control via diminishing step sizes is unnecessary; second, a form of ``weak exploration'' is automatically achieved through the stochastic gradient updates, since they prevent the action probabilities from decaying faster than $O(1/t)$, thus ensuring that every action is sampled infinitely often with probability $1$. These two findings can be used to show that the stochastic gradient update is already ``sufficient'' for bandits in the sense that exploration versus exploitation is automatically balanced in a manner that ensures almost sure convergence to a global optimum. These novel theoretical findings are further verified by experimental results.

Stochastic Gradient Succeeds for Bandits

TL;DR

This paper proves that the canonical stochastic gradient bandit algorithm achieves almost-sure global convergence to a globally optimal policy at an rate with a constant learning rate. The key innovations are a strong growth condition that makes the stochastic gradient noise naturally decay in tandem with progress, and an automatic weak exploration mechanism via the softmax Jacobian that prevents probabilities from collapsing too quickly. The authors develop non-uniform smoothness and Łojasiewicz-type tools to bound noise and establish both asymptotic convergence and finite-time rates, with experimental results corroborating the theory. The findings imply that simple gradient-based bandits can balance exploration and exploitation without decaying steps, offering practical benefits and potential extensions to broader reinforcement learning settings.

Abstract

We show that the \emph{stochastic gradient} bandit algorithm converges to a \emph{globally optimal} policy at an rate, even with a \emph{constant} step size. Remarkably, global convergence of the stochastic gradient bandit algorithm has not been previously established, even though it is an old algorithm known to be applicable to bandits. The new result is achieved by establishing two novel technical findings: first, the noise of the stochastic updates in the gradient bandit algorithm satisfies a strong ``growth condition'' property, where the variance diminishes whenever progress becomes small, implying that additional noise control via diminishing step sizes is unnecessary; second, a form of ``weak exploration'' is automatically achieved through the stochastic gradient updates, since they prevent the action probabilities from decaying faster than , thus ensuring that every action is sampled infinitely often with probability . These two findings can be used to show that the stochastic gradient update is already ``sufficient'' for bandits in the sense that exploration versus exploitation is automatically balanced in a manner that ensures almost sure convergence to a global optimum. These novel theoretical findings are further verified by experimental results.
Paper Structure (23 sections, 22 theorems, 201 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 23 sections, 22 theorems, 201 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Gradient Bandit Algorithms
  3. Preliminary Stochastic Gradient Analysis
  4. New Analysis: Noise Vanishes Automatically
  5. Non-uniform Smoothness: Landscape Properties
  6. Growth Conditions: Softmax Jacobian Behavior
  7. No Learning Rate Decay
  8. New Global Convergence Results
  9. Asymptotic Global Convergence
  10. Convergence Rate
  11. The Effect of Baselines
  12. Simulation Results
  13. Asymptotic Global Convergence
  14. Convergence Rate
  15. Average Gradient Norm Convergence
  16. ...and 8 more sections

Key Result

Proposition 2.3

alg:gradient_bandit_algorithm_sampled_reward is equivalent to the following stochastic gradient ascent update on $\pi_{\theta}^\top r$, where $\mathbb{E}_t{ [ \frac{d \pi_{\theta_t}^\top \hat{r}_t }{d \theta_t} ] } = \frac{d \pi_{\theta_t}^\top r}{d \theta_t }$, and $\left( \frac{d \pi_{\theta}}{d \theta} \right)^\top = \text{diag}{(\pi_{\theta})} - \pi_{\theta} \pi_{\theta}^\top$ is the Jacobian

Figures (4)

  • Figure 1: Visualization and intuition for \ref{['lem:strong_growth_conditions_sampled_reward']}. (a) Stochastic gradient scale. (b) True gradient norm. (c) The ratio of true gradient norm over $1$ minus largest action probability. Color bars contain minimum and maximum values of corresponding quantities.
  • Figure 2: Both subfigures show results for $\pi_{\theta_t}(a^*)$. The left is for $\pi_{\theta_1}(a^*) = 0.03$, and the right is for $\pi_{\theta_1}(a^*) = 0.02$.
  • Figure 3: Figure (a) shows the optimal action's probability and (b) shows $\log$ sub-optimal gap, which justifies our global convergence rate in \ref{['thm:convergence_rate_and_regret_gradient_bandit_sampled_reward']}.
  • Figure 4: Average squared gradient norm $\frac{1}{t} \cdot \sum\limits_{1 \le s \le t}{ \mathop{\mathrm{\mathbb{E}}}\limits{ [ \| \frac{d \pi_{\theta_s}^\top r}{d \theta_s} \|_2^2 ]} }$ in $\log$ scale (l.h.s. of \ref{['eq:almost_sure_convergence_gradient_bandit_algorithms_sampled_reward_intuition_1']}).

Theorems & Definitions (47)

  • Remark 2.2
  • Proposition 2.3
  • Proposition 3.1: Unbiased stochastic gradient with bounded variance / scale
  • Lemma 4.1: Non-uniform smoothness (NS), Lemma 2 in mei2021leveraging
  • Lemma 4.2: NS between iterates
  • Lemma 4.3: Strong growth condition; self-bounding noise property
  • Remark 4.4
  • Remark 4.5
  • Lemma 4.6: Constant learning rates
  • Corollary 4.7
  • ...and 37 more