Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lookahead identification in adversarial bandits: accuracy and memory bounds

Nataly Brukhim, Nicolò Cesa-Bianchi, Carlo Ciliberto

TL;DR

This work introduces lookahead identification, a task in which the goal of the learner is to select a future prediction window and commit in advance to an arm whose average reward over that window is within $\varepsilon$ of optimal, and investigates the role of memory.

Abstract

We study an identification problem in multi-armed bandits. In each round a learner selects one of $K$ arms and observes its reward, with the goal of eventually identifying an arm that will perform best at a {\it future} time. In adversarial environments, however, past performance may offer little information about the future, raising the question of whether meaningful identification is possible at all. In this work, we introduce \emph{lookahead identification}, a task in which the goal of the learner is to select a future prediction window and commit in advance to an arm whose average reward over that window is within $\varepsilon$ of optimal. Our analysis characterizes both the achievable accuracy of lookahead identification and the memory resources required to obtain it. From an accuracy standpoint, for any horizon $T$ we give an algorithm achieving $\varepsilon = O\bigl(1/\sqrt{\log T}\bigr)$ over $Ω(\sqrt{T})$ prediction windows. This demonstrates that, perhaps surprisingly, identification is possible in adversarial settings, despite significant lack of information. We also prove a near-matching lower bound showing that $\varepsilon = Ω\bigl(1/\log T\bigr)$ is unavoidable. We then turn to investigate the role of memory in our problem, first proving that any algorithm achieving nontrivial accuracy requires $Ω(K)$ bits of memory. Under a natural \emph{local sparsity} condition, we show that the same accuracy guarantees can be achieved using only poly-logarithmic memory.

Lookahead identification in adversarial bandits: accuracy and memory bounds

TL;DR

This work introduces lookahead identification, a task in which the goal of the learner is to select a future prediction window and commit in advance to an arm whose average reward over that window is within of optimal, and investigates the role of memory.

Abstract

We study an identification problem in multi-armed bandits. In each round a learner selects one of arms and observes its reward, with the goal of eventually identifying an arm that will perform best at a {\it future} time. In adversarial environments, however, past performance may offer little information about the future, raising the question of whether meaningful identification is possible at all. In this work, we introduce \emph{lookahead identification}, a task in which the goal of the learner is to select a future prediction window and commit in advance to an arm whose average reward over that window is within of optimal. Our analysis characterizes both the achievable accuracy of lookahead identification and the memory resources required to obtain it. From an accuracy standpoint, for any horizon we give an algorithm achieving over prediction windows. This demonstrates that, perhaps surprisingly, identification is possible in adversarial settings, despite significant lack of information. We also prove a near-matching lower bound showing that is unavoidable. We then turn to investigate the role of memory in our problem, first proving that any algorithm achieving nontrivial accuracy requires bits of memory. Under a natural \emph{local sparsity} condition, we show that the same accuracy guarantees can be achieved using only poly-logarithmic memory.
Paper Structure (16 sections, 7 theorems, 66 equations, 1 table, 3 algorithms)

This paper contains 16 sections, 7 theorems, 66 equations, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Memory constraints
  3. Prior work
  4. Best-arm identification with lookahead
  5. Memory bounded experts
  6. Memory bounded bandits
  7. Density prediction
  8. Lookahead BAI for multi-armed bandits
  9. Lower bounds
  10. Proof sketch
  11. Memory lower bound
  12. Lookahead BAI with bounded memory for sparse bandits
  13. Extension to real-valued rewards $X \in [0,1]^{K \times T}$
  14. Regret minimization for bandits with bounded memory
  15. Proof sketch
  16. ...and 1 more sections

Key Result

theorem 1

Let $K, T \in \mathbb{N}$ such that $K = \tilde{O}({{T^{1/4}}})$. For any bandit instance $X \in [0,1]^{K \times T}$, Algorithm alg:bandit_bai returns $t_0$, window size $w = \Omega(\sqrt{T})$, and arm $\hat{i}$ such that,

Theorems & Definitions (27)

  • definition 1: Lookahead BAI
  • theorem 1: Bandit lookahead BAI
  • lemma 1
  • proof
  • Claim 2
  • proof
  • theorem 3: Lower bound for lookahead BAI
  • Claim 4
  • Claim 5
  • theorem 6: Memory lower bound
  • ...and 17 more