Improved Algorithms for Stochastic Linear Bandits Using Tail Bounds for Martingale Mixtures

TL;DR

This work uses a novel tail bound for adaptive martingale mixtures to construct confidence sequences which are suitable for stochastic bandits and proves that a linear bandit algorithm based on these confidence sequences is guaranteed to achieve competitive worst-case regret.

Abstract

We present improved algorithms with worst-case regret guarantees for the stochastic linear bandit problem. The widely used "optimism in the face of uncertainty" principle reduces a stochastic bandit problem to the construction of a confidence sequence for the unknown reward function. The performance of the resulting bandit algorithm depends on the size of the confidence sequence, with smaller confidence sets yielding better empirical performance and stronger regret guarantees. In this work, we use a novel tail bound for adaptive martingale mixtures to construct confidence sequences which are suitable for stochastic bandits. These confidence sequences allow for efficient action selection via convex programming. We prove that a linear bandit algorithm based on our confidence sequences is guaranteed to achieve competitive worst-case regret. We show that our confidence sequences are tighter than competitors, both empirically and theoretically. Finally, we demonstrate that our tighter confidence sequences give improved performance in several hyperparameter tuning tasks.

Figures (6)

• Figure 1: Tighter upper and lower confidence bounds via tail bounds for martingale mixtures. The upper and lower confidence bounds of CMM-UCB (left), AMM-UCB (middle), and OFUL abbasi2011improved (right) for a test function linear in random Fourier features. The bounds from CMM-UCB and AMM-UCB are visibly closer to the true function (dashed line) than those of OFUL. The CMM-UCB confidence bounds are slightly tighter than the ones of AMM-UCB.
• Figure 2: Average confidence bound width for different data set sizes $T$ and feature dimensions $d$.
• Figure 3: The smoothed per-round expected reward (test accuracy) of our UCB algorithms compared with OFUL, IDS and Freq-TS in the SVM hyperparameter tuning experiments on three datasets. Shown is the mean reward over 100 runs of each experiment, after Gaussian kernel smoothing.
• Figure 4: The upper and lower confidence bounds of our CMM-UCB method (left) and Bayesian posterior credible intervals (right) with different choices of the prior. The top row uses the prior $\boldsymbol{f}_t \sim \mathcal{N}(\boldsymbol{0}, \Phi_t\Phi_t^{\top})$ for CMM-UCB and $\boldsymbol{\theta}^* \sim \mathcal{N}(\boldsymbol{0}, \mathbbm{1}))$ for Bayes. The middle row uses an informative prior: $\boldsymbol{f}_t \sim \mathcal{N}(\Phi_t\boldsymbol{\theta}^*, 0.1\Phi_t\Phi_t^{\top})$ for CMM-UCB and $\boldsymbol{\theta}^* \sim \mathcal{N}(\boldsymbol{\theta}^*, 0.1\mathbbm{1}))$ for Bayes. The bottom row uses a misspecified prior: $\boldsymbol{f}_t \sim \mathcal{N}(-\Phi_t\boldsymbol{\theta}^*, 0.1\Phi_t\Phi_t^{\top})$ for CMM-UCB and $\boldsymbol{\theta}^* \sim \mathcal{N}(-\boldsymbol{\theta}^*, 0.1\mathbbm{1}))$ for Bayes.
• Figure 5: The upper and lower confidence bounds of CMM-UCB with the standard (non-adaptive) sequence of Gaussian mixture distributions (purple) and the adaptive sequence of Gaussian mixture distributions (red).

Theorems & Definitions (45)

• Theorem 5.1: Tail Bound for Adaptive Martingale Mixtures
• Corollary 5.2: Martingale Mixture Confidence Sequence
• Theorem 6.1: Analytic UCB
• Theorem 7.5
• Theorem 7.6
• proof : Proof sketch
• Lemma A.1
• proof
• Lemma A.2
• proof