Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tangential Randomization in Linear Bandits (TRAiL): Guaranteed Inference and Regret Bounds

Arda Güçlü, Subhonmesh Bose

TL;DR

This work proves that any algorithm with an $\mathcal{O}(T^\alpha)$ expected regret growth must have an $\Omega(T^{1-\alpha})$ asymptotic growth in expected inference quality, and characterize an $\Omega(\sqrt{T})$ minimax lower bound for any algorithm on the expected regret that covers a wide variety of action/parameter sets and noise processes.

Abstract

We propose and analyze TRAiL (Tangential Randomization in Linear Bandits), a computationally efficient regret-optimal forced exploration algorithm for linear bandits on action sets that are sublevel sets of strongly convex functions. TRAiL estimates the governing parameter of the linear bandit problem through a standard regularized least squares and perturbs the reward-maximizing action corresponding to said point estimate along the tangent plane of the convex compact action set before projecting back to it. Exploiting concentration results for matrix martingales, we prove that TRAiL ensures a $Ω(\sqrt{T})$ growth in the inference quality, measured via the minimum eigenvalue of the design (regressor) matrix with high-probability over a $T$-length period. We build on this result to obtain an $\mathcal{O}(\sqrt{T} \log(T))$ upper bound on cumulative regret with probability at least $ 1 - 1/T$ over $T$ periods, and compare TRAiL to other popular algorithms for linear bandits. Then, we characterize an $Ω(\sqrt{T})$ minimax lower bound for any algorithm on the expected regret that covers a wide variety of action/parameter sets and noise processes. Our analysis not only expands the realm of lower-bounds in linear bandits significantly, but as a byproduct, yields a trade-off between regret and inference quality. Specifically, we prove that any algorithm with an $\mathcal{O}(T^α)$ expected regret growth must have an $Ω(T^{1-α})$ asymptotic growth in expected inference quality. Our experiments on the $L^p$ unit ball as action sets reveal how this relation can be violated, but only in the short-run, before returning to respect the bound asymptotically. In effect, regret-minimizing algorithms must have just the right rate of inference -- too fast or too slow inference will incur sub-optimal regret growth.

Tangential Randomization in Linear Bandits (TRAiL): Guaranteed Inference and Regret Bounds

TL;DR

This work proves that any algorithm with an expected regret growth must have an asymptotic growth in expected inference quality, and characterize an minimax lower bound for any algorithm on the expected regret that covers a wide variety of action/parameter sets and noise processes.

Abstract

We propose and analyze TRAiL (Tangential Randomization in Linear Bandits), a computationally efficient regret-optimal forced exploration algorithm for linear bandits on action sets that are sublevel sets of strongly convex functions. TRAiL estimates the governing parameter of the linear bandit problem through a standard regularized least squares and perturbs the reward-maximizing action corresponding to said point estimate along the tangent plane of the convex compact action set before projecting back to it. Exploiting concentration results for matrix martingales, we prove that TRAiL ensures a growth in the inference quality, measured via the minimum eigenvalue of the design (regressor) matrix with high-probability over a -length period. We build on this result to obtain an upper bound on cumulative regret with probability at least over periods, and compare TRAiL to other popular algorithms for linear bandits. Then, we characterize an minimax lower bound for any algorithm on the expected regret that covers a wide variety of action/parameter sets and noise processes. Our analysis not only expands the realm of lower-bounds in linear bandits significantly, but as a byproduct, yields a trade-off between regret and inference quality. Specifically, we prove that any algorithm with an expected regret growth must have an asymptotic growth in expected inference quality. Our experiments on the unit ball as action sets reveal how this relation can be violated, but only in the short-run, before returning to respect the bound asymptotically. In effect, regret-minimizing algorithms must have just the right rate of inference -- too fast or too slow inference will incur sub-optimal regret growth.

Paper Structure

This paper contains 16 sections, 15 theorems, 124 equations, 6 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Summary of Contributions
  3. Paper Organization
  4. The Algorithm
  5. Theoretical Results
  6. Geometric Properties of the Action Set
  7. Analysis of Guaranteed Inference
  8. Regret Analysis
  9. A Universal Lower-Bound on Regret
  10. Emprical Comparison with Other Algorithms
  11. Experiments on the Unit Sphere
  12. Experiments on Randomly Sampled Ellipsoids
  13. The Curious Case of the Lp Unit-Norm Balls
  14. Conclusions and Future Work
  15. Algorithms Compared in Section \ref{['sec:numerics']}
  16. ...and 1 more sections

Key Result

Lemma 1

Suppose Assumptions assumption_of_parameter_set and assumption_of_action_set hold. For all $\theta \in \Theta$, the reward-maximizing action $a^\star(\theta)= \mathop{\rm argmax}_{a\in{\cal A}} \ \theta^{\top} a$ is unique and $\nabla g(a^\star(\theta)) = \omega^\star(\theta ) \theta$ for some $\ome

Figures (6)

  • Figure 1: Visualization of action formation at time $t$ via $a_{t} = {\textrm{proj}_{{\cal A}}\left(a^\star(\widehat{\theta}_t) + \nu^2_t \mu^2_t\right)}$ for a 2-dimensional convex action set ${\cal A}$.
  • Figure 2: Comparing FEL, TS, UCB, and TRAiL on a spherical action set in $\mathbb{R}^{10}$ where the area that lies in one standard deviation from the means of the curves are shaded with their respective colors; (a) shows the effect of $D^{\textrm{TRAiL}}$ on regret, (b)-(c) plots regret with reward error variances of $0.1$ and $1$, respectively.
  • Figure 3: Comparing FEL, TS, and TRAiL on randomly selected ellipsoidal action sets where $\varepsilon_t \sim {\cal N}(0,0.1)$; (a), (b) show the regret curves for these algorithms where the the action sets are in $\mathbb{R}^{20}$ and $\mathbb{R}^{100}$, respectively, (c) provides a speed comparison in seconds with the dimension of the action space varied between 10 and 200 and averaged over 5 runs with $T = 10^4$.
  • Figure 4: The performance of BayesTS, given in linear_thompson_sampling_revisited, on ${\cal A}^{10}$ with $\sigma_\varepsilon^{2} = 0.1$ and $\sigma_{\theta^\star}^{2} = 0.01$, over 20 runs. (a) plots the progress of regret, (b) plots the progress of inference equality, measured via $\lambda_{\min}(V_t)$, (c) plots the log-ratio of regret and inference quality with time, and (d) shows the validity of \ref{['eq:er.ei.t']}.
  • Figure 5: Confidence ellipsoids, defined at Lemma \ref{['lemma:probability_Et']} with $a_{\max} = \sqrt{2}$, for the progress of BayesTS with $\theta^\star = (1,1)$ at ()$T_1=10^3$ iterations and ()$T_2=10^4$ iterations are overlaid on the heatmap of $\lambda_1(\nabla a^\star(\theta^\star))$.
  • ...and 1 more figures

Theorems & Definitions (15)

  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 6
  • Lemma 7
  • Lemma 8
  • Lemma 9
  • Lemma 10
  • ...and 5 more