Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits
Xuheng Li, Quanquan Gu
TL;DR
This work introduces FGTS-VA, a variance-aware Thompson Sampling algorithm for contextual bandits with general reward functions. By augmenting the FGTS posterior with variance-dependent weights and a feel-good exploration term, and analyzing it through a generalized decoupling coefficient, FGTS-VA achieves regret bounds that scale with the cumulative variance $\Lambda$ and model complexity $\operatorname{dc}_{\lambda,\epsilon,T}(\mathcal{F})$, matching minimax rates in linear settings. Specifically, $\mathbb{E}[\text{Regret}(T)] \lesssim \sqrt{(1+\Lambda) \operatorname{dc}_{\lambda,\epsilon,T}(\mathcal{F}) \log|\mathcal{F}|} + \operatorname{dc}_{\lambda,\epsilon,T}(\mathcal{F})$, and in contextual linear bandits this reduces to $\tilde{O}(d\sqrt{\Lambda}+d)$ with $\operatorname{dc}=\tilde{O}(d)$. The approach uses two key proof techniques to handle reward randomness and KL-regularization, and yields horizon-free guarantees for the FGTS framework under variance revealing, offering a viable, efficient alternative to UCB-based variance-aware methods.
Abstract
Variance-dependent regret bounds have received increasing attention in recent studies on contextual bandits. However, most of these studies are focused on upper confidence bound (UCB)-based bandit algorithms, while sampling based bandit algorithms such as Thompson sampling are still understudied. The only exception is the LinVDTS algorithm (Xu et al., 2023), which is limited to linear reward function and its regret bound is not optimal with respect to the model dimension. In this paper, we present FGTSVA, a variance-aware Thompson Sampling algorithm for contextual bandits with general reward function with optimal regret bound. At the core of our analysis is an extension of the decoupling coefficient, a technique commonly used in the analysis of Feel-good Thompson sampling (FGTS) that reflects the complexity of the model space. With the new decoupling coefficient denoted by $\mathrm{dc}$, FGTS-VA achieves the regret of $\tilde{O}(\sqrt{\mathrm{dc}\cdot\log|\mathcal{F}|\sum_{t=1}^Tσ_t^2}+\mathrm{dc})$, where $|\mathcal{F}|$ is the size of the model space, $T$ is the total number of rounds, and $σ_t^2$ is the subgaussian norm of the noise (e.g., variance when the noise is Gaussian) at round $t$. In the setting of contextual linear bandits, the regret bound of FGTSVA matches that of UCB-based algorithms using weighted linear regression (Zhou and Gu, 2022).