Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploration via linearly perturbed loss minimisation

David Janz, Shuai Liu, Alex Ayoub, Csaba Szepesvári

TL;DR

The paper addresses exploration in structured stochastic bandits by introducing EVILL, which perturbs the regularised loss with a random linear term and minimises the perturbed objective to select actions. It shows EVILL coincides with PHE in generalized linear bandits but remains well-defined and effective beyond that regime due to data-dependent perturbations guided by Fisher information, yielding Thompson-like regret guarantees. Theoretical results establish regret bounds under self-concordant NEF assumptions and concentration guarantees for NEFs, while experiments on logistic and Rayleigh bandits illustrate the method’s robustness and practical simplicity. EVILL thus provides a simple, plug-in approach that approximates Bayesian-style exploration with comparable performance and broader applicability, including potential use in reinforcement learning.

Abstract

We introduce exploration via linear loss perturbations (EVILL), a randomised exploration method for structured stochastic bandit problems that works by solving for the minimiser of a linearly perturbed regularised negative log-likelihood function. We show that, for the case of generalised linear bandits, EVILL reduces to perturbed history exploration (PHE), a method where exploration is done by training on randomly perturbed rewards. In doing so, we provide a simple and clean explanation of when and why random reward perturbations give rise to good bandit algorithms. We propose data-dependent perturbations not present in previous PHE-type methods that allow EVILL to match the performance of Thompson-sampling-style parameter-perturbation methods, both in theory and in practice. Moreover, we show an example outside generalised linear bandits where PHE leads to inconsistent estimates, and thus linear regret, while EVILL remains performant. Like PHE, EVILL can be implemented in just a few lines of code.

Exploration via linearly perturbed loss minimisation

TL;DR

The paper addresses exploration in structured stochastic bandits by introducing EVILL, which perturbs the regularised loss with a random linear term and minimises the perturbed objective to select actions. It shows EVILL coincides with PHE in generalized linear bandits but remains well-defined and effective beyond that regime due to data-dependent perturbations guided by Fisher information, yielding Thompson-like regret guarantees. Theoretical results establish regret bounds under self-concordant NEF assumptions and concentration guarantees for NEFs, while experiments on logistic and Rayleigh bandits illustrate the method’s robustness and practical simplicity. EVILL thus provides a simple, plug-in approach that approximates Bayesian-style exploration with comparable performance and broader applicability, including potential use in reinforcement learning.

Abstract

We introduce exploration via linear loss perturbations (EVILL), a randomised exploration method for structured stochastic bandit problems that works by solving for the minimiser of a linearly perturbed regularised negative log-likelihood function. We show that, for the case of generalised linear bandits, EVILL reduces to perturbed history exploration (PHE), a method where exploration is done by training on randomly perturbed rewards. In doing so, we provide a simple and clean explanation of when and why random reward perturbations give rise to good bandit algorithms. We propose data-dependent perturbations not present in previous PHE-type methods that allow EVILL to match the performance of Thompson-sampling-style parameter-perturbation methods, both in theory and in practice. Moreover, we show an example outside generalised linear bandits where PHE leads to inconsistent estimates, and thus linear regret, while EVILL remains performant. Like PHE, EVILL can be implemented in just a few lines of code.
Paper Structure (40 sections, 16 theorems, 155 equations, 2 figures)

This paper contains 40 sections, 16 theorems, 155 equations, 2 figures.

Table of Contents

  1. INTRODUCTION
  2. PROBLEM SETTING
  3. Notation
  4. Bandits & regret
  5. Regularised negative log-likelihood
  6. Examples
  7. THE EVILL ALGORITHM
  8. Why do linear perturbations work?
  9. Relation to Thompson sampling
  10. RESULTS FOR GENERALISED LINEAR BANDITS
  11. Natural exponential families
  12. The equivalence of EVILL and PHE
  13. Regret guarantees
  14. Confidence sets for NEFs
  15. EXPERIMENTS
  16. ...and 25 more sections

Key Result

Proposition 1

Let $\ell(x,y;\theta) = -\log p(y; \phi_x(\theta))$ and $\ell(x;\theta) = \int \ell(x,y;\theta) P(dy; \phi_x(\theta_\star))$. Then, under suitable regularity assumptions,

Figures (2)

  • Figure 1: Regret of TSL, EVILL and FPL on two logistic linear bandit tasks: high variance, where $\dot\mu(x_\star^\top\theta_\star)$ is high, and low variance, where $\dot\mu(x_\star^\top\theta_\star)$ is small. Box plots are based on the regret from 100 instances, and show the median and the interquartile (IQ) range, with whiskers restricted to $1.5\times$ the IQ range.
  • Figure 2: Rayleigh parameter estimation and bandit experiments. The left panel shows the $\ell_2$ error in estimating the parameter of a Rayleigh bandit when data is i.i.d., uncontrolled by the bandit algorithm. In addition to PHE and EVILL's estimates, the MLE estimate using the same data is also shown for reference. The right panel shows regret for TSL, PHE and EVILL for the same Rayleigh bandit problem, over $100$ steps of interaction. Both panels plot means over $100$ iterations and no confidence intervals---the latter were too tight to be visible. EVILL and TSL lines in right panel overlap.

Theorems & Definitions (59)

  • Example 1: Generalised linear bandits filippi2010parametric
  • Example 2: Logistic linear bandits
  • Example 3: Linear bandits with Gaussian rewards
  • Proposition 1
  • Proposition 2
  • Example 4: Rayleigh linear bandit
  • Remark 1
  • Theorem 1
  • Remark 2
  • Lemma 1
  • ...and 49 more