Linear bandits with polylogarithmic minimax regret

TL;DR

This work introduces a linear bandit model with vanishing noise, where the noise variance satisfies $\sigma_t^2 \le 1-\langle\theta,a_t\rangle^2$, and proposes LinUCB-VN, a weighted regularized least-squares approach that preserves optimism while adapting to decreasing noise. A geometric action-scheme ensures the design matrix satisfies $\lambda_{\min}(V_t)=\Omega(\sqrt{\lambda_{\max}(V_t)})$, enabling instantaneous regret $\sim 1/t$ and a cumulative polylogarithmic regret of $O(d^4\log^3 T)$. The analysis combines a weighted confidence region, a batch-based action update, and a novel eigenvalue-growth argument that is robust to the noise model. The paper also establishes a minimax lower bound for the constant-noise setting and discusses the limitations of standard lower-bound techniques under vanishing noise, underscoring the novelty and potential of the proposed approach in quantum tomography and other applications where measurement noise decays with alignment to the unknown parameter.

Abstract

We study a noise model for linear stochastic bandits for which the subgaussian noise parameter vanishes linearly as we select actions on the unit sphere closer and closer to the unknown vector. We introduce an algorithm for this problem that exhibits a minimax regret scaling as $\log^3(T)$ in the time horizon $T$, in stark contrast the square root scaling of this regret for typical bandit algorithms. Our strategy, based on weighted least-squares estimation, achieves the eigenvalue relation $λ_{\min} ( V_t ) = Ω(\sqrt{λ_{\max}(V_t ) })$ for the design matrix $V_t$ at each time step $t$ through geometrical arguments that are independent of the noise model and might be of independent interest. This allows us to tightly control the expected regret in each time step to be of the order $O(\frac1{t})$, leading to the logarithmic scaling of the cumulative regret.

Figures (3)

• Figure 1: Scheme for the choice of actions $a_t^+,a_t^-$ of LinUCB-VN. The actions are selected as the projections of the extremal points across the largest axis of the confidence region centered around a weighted least squares estimator $\widetilde{\theta}^{\text{w}}_t$ of the unknown parameter $\theta$. This choice is sufficient to increase the minimum eigenvalue of $V_t$ such that the relation $\lambda_{\min}(V_t) = \Omega (\sqrt{\lambda_{\max}(V_t)})$ is satisfied. Moreover, the actions $a_t^+$ and $a_t^-$ are sufficiently close to $\theta$ to keep the regret small.
• Figure 2: We numerically test LinUCB and LinUCB-VN in a linear bandit with action set $\mathcal{A} = \mathbb{S}^2$ and reward model $r_t = \mathcal{N}(\langle \theta , a_t \rangle , 1 - \langle \theta , a_t \rangle^2 )$. Each point in the graphic is run independently and averaged over 100 instances for random environments $\theta\in\mathbb{S}^2$. Left plot: Scaling of the regret for LinUCB algorithm and LinUCB-VN. We fit the functions $R(t) = 1.86\log^2 t$ for LinUCB-VN and $R(t) = 0.88\sqrt{t\log t}$ for LinUCB. Right plots: Scaling of the maximum and minimum eigenvalue of the matrix $V_t$ for LinUCB-VN. The scaling shows the relation $\lambda_{\min} = \Omega ( \sqrt{\lambda_{\max}} )$. We fit the function $\lambda_{\min}(V_t) = 0.2059t$ for the minimum eigenvalue and $\lambda_{\max}(V_t) = 0.0012t^2$ for the maximum eigenvalue. The behavior $\lambda_{\min}(V_t) = \Theta ( t )$ is the one that gives us the theoretical guarantee of polylogarithmic scaling of the regret.
• Figure 3: Sketch for the triangle inequality used to bound $\| \theta - a^\pm_{t,i} \|_2$ in \ref{['eq:theta_at_bound']}. The red lines represent the distances $(i),(ii)$ and $(iii)$. Under the event $\theta\in\mathcal{C}_{t-1}$ we can use $\mathcal{C}_{t-1} \subseteq \mathbb{B}^d_r (\tilde{\theta}^w_{t-1} )$ with $r$ being the longest axis of the ellipsoid and bound all distances by the diameter $2r$.

Theorems & Definitions (12)

• Theorem 1: main result, informal version
• Lemma 2
• Theorem 3
• Theorem 4
• Theorem 6: Theorem 3.9 in lattimore_szepesvári_2020
• Corollary 7: Corollary III.1.2 in bhatia97
• Theorem 8
• Lemma 9
• Theorem 10: MichelPetrovitch1901
• Corollary 11