Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits

Ha Manh Bui, Enrique Mallada, Anqi Liu

TL;DR

This work proposes Neural-$\sigma^2$-LinearUCB, a variance-aware algorithm that utilizes $\sigma^2_t$, an upper bound of the reward noise variance at round $t, to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement.

Abstract

By leveraging the representation power of deep neural networks, neural upper confidence bound (UCB) algorithms have shown success in contextual bandits. To further balance the exploration and exploitation, we propose Neural-$σ^2$-LinearUCB, a variance-aware algorithm that utilizes $σ^2_t$, i.e., an upper bound of the reward noise variance at round $t$, to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement. We provide an oracle version for our algorithm characterized by an oracle variance upper bound $σ^2_t$ and a practical version with a novel estimation for this variance bound. Theoretically, we provide rigorous regret analysis for both versions and prove that our oracle algorithm achieves a better regret guarantee than other neural-UCB algorithms in the neural contextual bandits setting. Empirically, our practical method enjoys a similar computational efficiency, while outperforming state-of-the-art techniques by having a better calibration and lower regret across multiple standard settings, including on the synthetic, UCI, MNIST, and CIFAR-10 datasets.

Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits

TL;DR

This work proposes Neural--LinearUCB, a variance-aware algorithm that utilizes , an upper bound of the reward noise variance at round $t, to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement.

Abstract

By leveraging the representation power of deep neural networks, neural upper confidence bound (UCB) algorithms have shown success in contextual bandits. To further balance the exploration and exploitation, we propose Neural--LinearUCB, a variance-aware algorithm that utilizes , i.e., an upper bound of the reward noise variance at round , to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement. We provide an oracle version for our algorithm characterized by an oracle variance upper bound and a practical version with a novel estimation for this variance bound. Theoretically, we provide rigorous regret analysis for both versions and prove that our oracle algorithm achieves a better regret guarantee than other neural-UCB algorithms in the neural contextual bandits setting. Empirically, our practical method enjoys a similar computational efficiency, while outperforming state-of-the-art techniques by having a better calibration and lower regret across multiple standard settings, including on the synthetic, UCI, MNIST, and CIFAR-10 datasets.

Paper Structure

This paper contains 31 sections, 12 theorems, 82 equations, 18 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Problem setting
  4. Neural Linear Upper Confidence Bound
  5. Neural Variance-Aware Linear Upper Confidence Bound Algorithm
  6. Oracle algorithm
  7. Practical algorithm
  8. Theoretical analysis
  9. Experiments
  10. Synthetic datasets
  11. Real-world datasets
  12. Uncertainty estimation evaluations
  13. Computational efficiency evaluations
  14. Ablation study for ours
  15. Related work
  16. ...and 16 more sections

Key Result

Theorem 3.2

The reward r.v. $r_{t,a_t}$ in Eq. eq:reward_gen has the true mean $\mathbb{E}[r_{t,a_t}] = h(\mathbf{x}_{t,a_t})$ and variance $Var(r_{t,a_t}) = \mathbb{E}[\xi^2_t\mid \mathbf{x}_{1:t, a_{1:t}}, \xi_{1:t-1}]$. The proof is in Apd. proof:thm:reward_distribution.

Figures (18)

  • Figure 1: Cumulative regret results on the synthetic data across 10 runs with different seeds. More baselines for comparison and a zoom-in figure are in Fig. \ref{['fig:full_demo']}. A short demo is available at https://colab.research.google.com/drive/1WmuMocSfIxtfzSrdSZvVRgYNaM6ePgNN?usp=sharing.
  • Figure 2: Our ours can be more uncertain (i.e., more exploration) if the reward noise $Var(\xi_t)$ is high, and more certain (i.e., more exploitation) if $Var(\xi_t)$ is small.
  • Figure 3: Cumulative regret results on the real-world data across 10 runs with different seeds.
  • Figure 4: (a) Visualization of calibration error in Eq. \ref{['eq:calib']} with reliability diagram on $h_1(\mathbf{x}_{t,a})$ dataset. (b) Reward variance $Var(r_t)$, our estimation for the variance upper bound $\sigma_t^2$, and the upper bound $R^2$ comparison.
  • Figure 5: (a) Comparison in cumulative regret between different settings. (b) Variance estimation error between MLE and MSE settings of our ours.
  • ...and 13 more figures

Theorems & Definitions (23)

  • Remark 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Remark 4.1
  • Definition 4.3
  • Theorem 4.5
  • Corollary 4.6
  • Remark 4.7
  • Theorem 4.8
  • Remark 4.9
  • ...and 13 more