Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eluder dimension: localise it!

Alireza Bakhtiari, Alex Ayoub, Samuel Robertson, David Janz, Csaba Szepesvári

TL;DR

A localisation method is introduced for the eluder dimension of generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret bounds, and allows for the first genuine first-order bounds for finite-horizon reinforcement learning tasks with bounded cumulative returns.

Abstract

We establish a lower bound on the eluder dimension of generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret bounds. To address this, we introduce a localisation method for the eluder dimension; our analysis immediately recovers and improves on classic results for Bernoulli bandits, and allows for the first genuine first-order bounds for finite-horizon reinforcement learning tasks with bounded cumulative returns.

Eluder dimension: localise it!

TL;DR

A localisation method is introduced for the eluder dimension of generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret bounds, and allows for the first genuine first-order bounds for finite-horizon reinforcement learning tasks with bounded cumulative returns.

Abstract

We establish a lower bound on the eluder dimension of generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret bounds. To address this, we introduce a localisation method for the eluder dimension; our analysis immediately recovers and improves on classic results for Bernoulli bandits, and allows for the first genuine first-order bounds for finite-horizon reinforcement learning tasks with bounded cumulative returns.
Paper Structure (32 sections, 40 theorems, 183 equations, 2 algorithms)

This paper contains 32 sections, 40 theorems, 183 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Background on generalised linear models & loss functions
  4. Bandits with bounded costs and the $\ell$-UCB algorithm
  5. Algorithm
  6. The localised eluder dimension & first-order regret bounds for bandits
  7. Why localisation matters: eluder dimension lower bound for generalised linear models
  8. Regret upper bound with localisation for the generalised linear model setting
  9. Discussion in the logistic bandit & maximum likelihood estimation settings
  10. First-order regret bounds for online reinforcement learning
  11. Preliminaries on Q-functions, Bellman optimality operators and greedy policies
  12. The $\ell$-GOLF algorithm, model and loss assumptions & regret bound
  13. Conclusion
  14. Self-concordance & convex relaxation
  15. A uniform Bernstein concentration inequality
  16. ...and 17 more sections

Key Result

theorem 1

Fix $\delta \in (0,1)$, $n \in \mathbf{N}_+$, bandit instance $\mathcal{P}$, model class $\mathcal{F}$ and a loss function $\ell$. Suppose that $(\mathcal{P}, \mathcal{F}, \ell)$ satisfy ass:bounded-costass:realass:loss. Let $N_n$ denote the $1/n$-covering number of $\Phi(\mathcal{F})$ with respect Let $\mathcal{F}^\prime \subset \mathcal{F}$, and denote by $d_n$ the $1/n$-eluder dimension of $\b

Theorems & Definitions (82)

  • remark 1
  • example 1
  • example 2
  • definition 1
  • definition 2: Eluder dimension
  • remark 2
  • theorem 1: Regret bound for $\ell$-UCB in bandits
  • remark 3
  • remark 4
  • theorem 2: name=GLM $\ell_1$-eluder dimension lower bound,restate=eluderLowerThm
  • ...and 72 more