Geometry-Aware Approaches for Balancing Performance and Theoretical Guarantees in Linear Bandits

TL;DR

This work addresses the gap between strong empirical performance and weak frequentist guarantees for linear bandit algorithms like Thompson sampling and Greedy. It develops a data-driven, geometry-aware framework (POFUL) that leverages the full $d$-dimensional confidence ellipsoid around the unknown parameter $\theta^*$ to derive practical regret bounds and to enable course-correction. By introducing a data-driven regret proxy and an adaptive meta-algorithm (TS-MR/Greedy-MR), the paper achieves minimax frequentist regret $\tilde{O}(d\sqrt{T})$ while preserving the empirical strengths of the base algorithms. Through synthetic and real-world experiments, the proposed approach demonstrates robust performance benefits and practical applicability, bridging theory and practice in linear bandits.

Abstract

This paper is motivated by recent research in the $d$-dimensional stochastic linear bandit literature, which has revealed an unsettling discrepancy: algorithms like Thompson sampling and Greedy demonstrate promising empirical performance, yet this contrasts with their pessimistic theoretical regret bounds. The challenge arises from the fact that while these algorithms may perform poorly in certain problem instances, they generally excel in typical instances. To address this, we propose a new data-driven technique that tracks the geometric properties of the uncertainty ellipsoid around the main problem parameter. This methodology enables us to formulate a data-driven frequentist regret bound, which incorporates the geometric information, for a broad class of base algorithms, including Greedy, OFUL, and Thompson sampling. This result allows us to identify and ``course-correct" problem instances in which the base algorithms perform poorly. The course-corrected algorithms achieve the minimax optimal regret of order $\tilde{\mathcal{O}}(d\sqrt{T})$ for a $T$-period decision-making scenario, effectively maintaining the desirable attributes of the base algorithms, including their empirical efficacy. We present simulation results to validate our findings using synthetic and real data.

Figures (5)

• Figure 1: (a) POFUL algorithms illustration for general $\iota_t$ and $\tau_t$. (b) Special cases: OFUL ($\iota_t = 0$, $\tau_t = 1$), TS ($\tau_t = 0$), and Greedy ($\iota_t = \tau_t = 0$).
• Figure 2: Illustration of potentially optimal actions set $\mathcal{C}_t$ in $\mathbb{R}^2$. (a): $\mathcal{C}_t$ is $\mathcal{E}_t$'s projection onto ${\mathcal{S}}_{d-1}$. (b): As more data is collected, $\mathcal{E}_t$ shrinks (colors show exploration levels). Potentially optimal actions point in similar directions, determining their $V_t$-norm. This suggests their $V_t$-norm range could be estimated geometrically.
• Figure 3: Simulation results on synthetic data. (a) - (c): Cumulative regret of TS-MR and Greedy-MR versus baseline algorithms. Shaded regions show $\pm 2$ SE of mean regret. (d) - (f): Fraction of OFUL actions in TS-MR and Greedy-MR.
• Figure 4: Simulation results on real-world datasets. (a) - (c): Cumulative regret of all algorithms. Shaded regions show the $\pm 2$ SE of the mean regret. (d) - (f): Fraction of OFUL actions of TS-MR and Greedy-MR.
• Figure 5: Illustration of Step 1 and 2 in $\mathbb{R}^2$. Orange dashed rays: rays starting from the origin might have different numbers of intersections with $\mathcal{E}_t$, indicating whether the corresponding action lies in the projection of $\mathcal{E}_t$ onto ${\mathcal{S}}_{d-1}$. Blue dashed curve: the ellipsoid with fixed $V_t$-norm $\left\{\theta:\|\theta\|_{V_t}=\phi_t\right\}$. The intersection of this ellipsoid and $\mathcal{E}_t$ has the same projection as $\mathcal{E}_t$ onto ${\mathcal{S}}_{d-1}$.

Theorems & Definitions (22)

• Proposition 1: Theorem 2 in abbasi2011improved
• Definition 1
• Proposition 2: Lemma 11 in abbasi2011improved
• Definition 2
• Example 1: OFUL
• Definition 3
• Example 2: TS
• Example 3: Greedy
• Definition 4
• Proposition 3