Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear Bandits with Non-i.i.d. Noise

Baptiste Abélès, Eugenio Clerico, Hamish Flynn, Gergely Neu

TL;DR

This paper addresses linear bandits under non-i.i.d. noise by introducing mixing sub-Gaussian noise and developing confidence sequences via an online-to-confidence-set reduction. It proposes Mixing-LinUCB, which uses delayed feedback and ellipsoidal confidence sets to accommodate dependencies, yielding regret bounds that depend on the decay rate of the noise's dependence (geometric or algebraic mixing). The results show that, up to a mixing-time factor, the standard i.i.d. rates are recovered for geometrically mixing noise, and they extend to sublinear rates for slower decays, with both worst-case and gap-dependent guarantees. The work provides a principled approach to learning under temporally dependent noise and outlines practical directions for learning the mixing structure and relaxing oblivious-adversary assumptions.

Abstract

We study the linear stochastic bandit problem, relaxing the standard i.i.d. assumption on the observation noise. As an alternative to this restrictive assumption, we allow the noise terms across rounds to be sub-Gaussian but interdependent, with dependencies that decay over time. To address this setting, we develop new confidence sequences using a recently introduced reduction scheme to sequential probability assignment, and use these to derive a bandit algorithm based on the principle of optimism in the face of uncertainty. We provide regret bounds for the resulting algorithm, expressed in terms of the decay rate of the strength of dependence between observations. Among other results, we show that our bounds recover the standard rates up to a factor of the mixing time for geometrically mixing observation noise.

Linear Bandits with Non-i.i.d. Noise

TL;DR

This paper addresses linear bandits under non-i.i.d. noise by introducing mixing sub-Gaussian noise and developing confidence sequences via an online-to-confidence-set reduction. It proposes Mixing-LinUCB, which uses delayed feedback and ellipsoidal confidence sets to accommodate dependencies, yielding regret bounds that depend on the decay rate of the noise's dependence (geometric or algebraic mixing). The results show that, up to a mixing-time factor, the standard i.i.d. rates are recovered for geometrically mixing noise, and they extend to sublinear rates for slower decays, with both worst-case and gap-dependent guarantees. The work provides a principled approach to learning under temporally dependent noise and outlines practical directions for learning the mixing structure and relaxing oblivious-adversary assumptions.

Abstract

We study the linear stochastic bandit problem, relaxing the standard i.i.d. assumption on the observation noise. As an alternative to this restrictive assumption, we allow the noise terms across rounds to be sub-Gaussian but interdependent, with dependencies that decay over time. To address this setting, we develop new confidence sequences using a recently introduced reduction scheme to sequential probability assignment, and use these to derive a bandit algorithm based on the principle of optimism in the face of uncertainty. We provide regret bounds for the resulting algorithm, expressed in terms of the decay rate of the strength of dependence between observations. Among other results, we show that our bounds recover the standard rates up to a factor of the mixing time for geometrically mixing observation noise.

Paper Structure

This paper contains 18 sections, 15 theorems, 57 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation.
  3. Preliminaries on linear bandits
  4. Linear bandits with non-i.i.d. observation noise
  5. Constructing the confidence sequence
  6. Online-to-confidence set conversion for i.i.d. data
  7. Confidence sequence under mixing sub-Gaussian noise
  8. Regret bounds for Mixing-LinUCB
  9. Worst-case regret bounds
  10. Gap-dependent regret bounds
  11. Conclusion
  12. Technical Appendices and Supplementary Material
  13. Proof of Proposition \ref{['pro:uniform_ewa_regret']}
  14. Proof of Lemma \ref{['lemma:conc']}
  15. Proof of Lemma \ref{['lemma:reg']}
  16. ...and 3 more sections

Key Result

Proposition 1

For some given $\phi$, let the noise satisfy Assumption ass:mixing-subgaussianity with $\sigma=1$. Fix $\delta\in(0,1)$, $\lambda>0$, and $d\geq 1$. For $t\geq 1$ let where $V_t = \sum_{s=1}^t X_tX_t^\top + \lambda\mathrm{Id}$, and $\widehat{\theta}_t = \mathop{\hbox{arg,min}}_{\theta\in\mathcal{B}(B)} \sum_{s=1}^t(\langle \theta,X_t\rangle -Y_t)^2$. Then, $(\mathcal{C}_t)_{t\geq 1}$ is an anytim

Theorems & Definitions (18)

  • Proposition 1
  • Proposition 2
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Lemma 3
  • Theorem 2
  • ...and 8 more