Stochastic Linear Bandits with Parameter Noise

TL;DR

This work studies stochastic linear bandits with parameter noise, where rewards follow $X_t = a_t^T \theta_t$ with $\theta_t$ drawn iid from a distribution having mean $\theta^*$ and covariance $\Sigma$. The authors develop variance-aware strategies that achieve minimax-optimal regret bounds for both finite action sets and $\ell_p$ unit balls with $p\le 2$, and establish matching lower bounds that depend on variance quantities $\sigma^2$ or $\sigma_q^2$. They introduce two algorithms: VASE, a variance-aware design-based elimination method for finite action sets, and VALEE, a simple adaptive explore-exploit algorithm for $\ell_p$ unit balls that attains near-optimal regret in both known and unknown covariance settings. The results reveal that the parameter-noise model can yield significantly better regret than the adversarial/aditive-noise settings for certain action geometries, and they provide variance-dependent bounds that generalize prior lower bounds to general covariance structures. The discussion highlights remaining challenges, such as achieving optimal rates when $\Sigma$ is unknown for $p>2$, and suggests directions for future improvement and application of variance-aware linear bandits.

Abstract

We study the stochastic linear bandits with parameter noise model, in which the reward of action $a$ is $a^\top θ$ where $θ$ is sampled i.i.d. We show a regret upper bound of $\widetilde{O} (\sqrt{d T \log (K/δ) σ^2_{\max})}$ for a horizon $T$, general action set of size $K$ of dimension $d$, and where $σ^2_{\max}$ is the maximal variance of the reward for any action. We further provide a lower bound of $\widetildeΩ (d \sqrt{T σ^2_{\max}})$ which is tight (up to logarithmic factors) whenever $\log (K) \approx d$. For more specific action sets, $\ell_p$ unit balls with $p \leq 2$ and dual norm $q$, we show that the minimax regret is $\widetildeΘ (\sqrt{dT σ^2_q)}$, where $σ^2_q$ is a variance-dependent quantity that is always at most $4$. This is in contrast to the minimax regret attainable for such sets in the classic additive noise model, where the regret is of order $d \sqrt{T}$. Surprisingly, we show that this optimal (up to logarithmic factors) regret bound is attainable using a very simple explore-exploit algorithm.

Theorems & Definitions (82)

• Theorem 3.1
• Remark 3.2
• Corollary 3.3
• Remark 3.4
• Lemma 3.5
• Lemma 3.6
• Theorem 3.7
• Remark 3.8
• Theorem 3.9
• Lemma 3.10