Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Truncated LinUCB for Stochastic Linear Bandits

Yanglei Song, Meng zhou

TL;DR

The paper investigates stochastic contextual linear bandits with a fixed number of arms and i.i.d. $d$-dimensional contexts, where rewards are linear in both arm parameters and contexts. It demonstrates that LinUCB over-explores, yielding suboptimal regret in both $d$ and $T$, and introduces Tr-LinUCB, which truncates exploration at time $S$ and then exploits, achieving $O(d\log T)$ regret when $S = C d \log T$ and a matching lower bound in a low-dimensional regime. Crucially, even if $S$ overshoots, the regret degrades by at most a $\log\log T$ factor independent of $d$, indicating practical robustness. The authors also prove a minimax lower bound showing $\Omega(d\log T)$ regret for certain problem families and validate their theory with extensive synthetic and real-data experiments, illustrating the method’s effectiveness and parameter-insensitivity. These results establish Tr-LinUCB as rate-optimal in the low-dimensional setting and offer a robust alternative to traditional exploration-heavy strategies.

Abstract

This paper considers contextual bandits with a finite number of arms, where the contexts are independent and identically distributed $d$-dimensional random vectors, and the expected rewards are linear in both the arm parameters and contexts. The LinUCB algorithm, which is near minimax optimal for related linear bandits, is shown to have a cumulative regret that is suboptimal in both the dimension $d$ and time horizon $T$, due to its over-exploration. A truncated version of LinUCB is proposed and termed "Tr-LinUCB", which follows LinUCB up to a truncation time $S$ and performs pure exploitation afterwards. The Tr-LinUCB algorithm is shown to achieve $O(d\log(T))$ regret if $S = Cd\log(T)$ for a sufficiently large constant $C$, and a matching lower bound is established, which shows the rate optimality of Tr-LinUCB in both $d$ and $T$ under a low dimensional regime. Further, if $S = d\log^κ(T)$ for some $κ>1$, the loss compared to the optimal is a multiplicative $\log\log(T)$ factor, which does not depend on $d$. This insensitivity to overshooting in choosing the truncation time of Tr-LinUCB is of practical importance.

Truncated LinUCB for Stochastic Linear Bandits

TL;DR

The paper investigates stochastic contextual linear bandits with a fixed number of arms and i.i.d. -dimensional contexts, where rewards are linear in both arm parameters and contexts. It demonstrates that LinUCB over-explores, yielding suboptimal regret in both and , and introduces Tr-LinUCB, which truncates exploration at time and then exploits, achieving regret when and a matching lower bound in a low-dimensional regime. Crucially, even if overshoots, the regret degrades by at most a factor independent of , indicating practical robustness. The authors also prove a minimax lower bound showing regret for certain problem families and validate their theory with extensive synthetic and real-data experiments, illustrating the method’s effectiveness and parameter-insensitivity. These results establish Tr-LinUCB as rate-optimal in the low-dimensional setting and offer a robust alternative to traditional exploration-heavy strategies.

Abstract

This paper considers contextual bandits with a finite number of arms, where the contexts are independent and identically distributed -dimensional random vectors, and the expected rewards are linear in both the arm parameters and contexts. The LinUCB algorithm, which is near minimax optimal for related linear bandits, is shown to have a cumulative regret that is suboptimal in both the dimension and time horizon , due to its over-exploration. A truncated version of LinUCB is proposed and termed "Tr-LinUCB", which follows LinUCB up to a truncation time and performs pure exploitation afterwards. The Tr-LinUCB algorithm is shown to achieve regret if for a sufficiently large constant , and a matching lower bound is established, which shows the rate optimality of Tr-LinUCB in both and under a low dimensional regime. Further, if for some , the loss compared to the optimal is a multiplicative factor, which does not depend on . This insensitivity to overshooting in choosing the truncation time of Tr-LinUCB is of practical importance.
Paper Structure (37 sections, 32 theorems, 103 equations, 3 figures, 3 tables)

This paper contains 37 sections, 32 theorems, 103 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our contributions
  3. More on stochastic linear bandits
  4. Outline and notations
  5. Problem Formulation and Tr-LinUCB Algorithm
  6. The proposed Tr-LinUCB Algorithm
  7. Regret analysis for Tr-LinUCB
  8. Assumptions
  9. Regret analysis without the condition \ref{['cond:unit_sphere']}
  10. Optimal dependence on the dimension $d$
  11. Sub-optimality of LinUCB
  12. Discrete components in contexts
  13. Experiments
  14. Synthetic Data
  15. Sensitivity Analysis on Synthetic Data
  16. ...and 22 more sections

Key Result

Lemma 2.1

Assume the condition assumption_parameter_noise holds. With probability at least $1-K/T$, $\|\hat{\boldsymbol{\theta}}_{t}^{(k)} - \boldsymbol{\theta}_{k}\|_{\mathbb{V}_{t}^{(k)}} \leqslant \sqrt{\beta_t^{(k)}}$ for all $t \in [T]$ and $k \in [K]$.

Figures (3)

  • Figure 1: Above the time axis are orders of the expected regret at time $t$ within each stage, where $\upsilon_{d,t} = d\log(T) + d^2 \log(t)$ and $\Upsilon_{d,T}$ in \ref{['chi_order']}, and below are important moments for the proposed Tr-LinUCB algorithm.
  • Figure 2: Cumulative regrets for different algorithms
  • Figure 3: Sensitivity analysis of the tuning parameter $\kappa$ in Tr-LinUCB on datasets (1)–(4).

Theorems & Definitions (78)

  • Lemma 2.1
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • Remark 1
  • Remark 2
  • Theorem 3.3
  • proof
  • Corollary 3.4
  • ...and 68 more