Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits

Yunbei Xu, Assaf Zeevi

TL;DR

This work introduces Upper Counterfactual Confidence Bounds (UCCB), a principled optimistic framework for contextual bandits with general function classes and large context spaces, using confidence bounds in policy space rather than action space. It presents two equivalent viewpoints—implicit policy-space UCB and explicit counterfactual-action bounds—and a practical algorithm with a regret of $\tilde{O}(\sqrt{KT\log|\mathcal{F}|})$ under realizability with an offline regression oracle. The authors develop a contextual potential function perspective and relaxation techniques to enable efficient computation, and extend the approach to infinite-action settings via counterfactual action divergence and average decision entropy, yielding first offline-regression-oracle solutions for several infinite-action models. They illustrate the framework across finite-action, infinite-action, and heterogeneous-action examples, including linear and generalized linear action models, and show how to generalize FALCON through optimistic subroutines. The work has potential implications for scalable, optimization-friendly reinforcement learning with rich function approximators and large or continuous action spaces.

Abstract

The principle of optimism in the face of uncertainty is one of the most widely used and successful ideas in multi-armed bandits and reinforcement learning. However, existing optimistic algorithms (primarily UCB and its variants) often struggle to deal with general function classes and large context spaces. In this paper, we study general contextual bandits with an offline regression oracle and propose a simple, generic principle to design optimistic algorithms, dubbed "Upper Counterfactual Confidence Bounds" (UCCB). The key innovation of UCCB is building confidence bounds in policy space, rather than in action space as is done in UCB. We demonstrate that these algorithms are provably optimal and computationally efficient in handling general function classes and large context spaces. Furthermore, we illustrate that the UCCB principle can be seamlessly extended to infinite-action general contextual bandits, provide the first solutions to these settings when employing an offline regression oracle.

Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits

TL;DR

This work introduces Upper Counterfactual Confidence Bounds (UCCB), a principled optimistic framework for contextual bandits with general function classes and large context spaces, using confidence bounds in policy space rather than action space. It presents two equivalent viewpoints—implicit policy-space UCB and explicit counterfactual-action bounds—and a practical algorithm with a regret of under realizability with an offline regression oracle. The authors develop a contextual potential function perspective and relaxation techniques to enable efficient computation, and extend the approach to infinite-action settings via counterfactual action divergence and average decision entropy, yielding first offline-regression-oracle solutions for several infinite-action models. They illustrate the framework across finite-action, infinite-action, and heterogeneous-action examples, including linear and generalized linear action models, and show how to generalize FALCON through optimistic subroutines. The work has potential implications for scalable, optimization-friendly reinforcement learning with rich function approximators and large or continuous action spaces.

Abstract

The principle of optimism in the face of uncertainty is one of the most widely used and successful ideas in multi-armed bandits and reinforcement learning. However, existing optimistic algorithms (primarily UCB and its variants) often struggle to deal with general function classes and large context spaces. In this paper, we study general contextual bandits with an offline regression oracle and propose a simple, generic principle to design optimistic algorithms, dubbed "Upper Counterfactual Confidence Bounds" (UCCB). The key innovation of UCCB is building confidence bounds in policy space, rather than in action space as is done in UCB. We demonstrate that these algorithms are provably optimal and computationally efficient in handling general function classes and large context spaces. Furthermore, we illustrate that the UCCB principle can be seamlessly extended to infinite-action general contextual bandits, provide the first solutions to these settings when employing an offline regression oracle.

Paper Structure

This paper contains 54 sections, 20 theorems, 111 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Contributions
  4. Systematic analysis of confidence bounds in policy space:
  5. Perspective of contextual potential function:
  6. Novel framework for infinite-action contextual bandits:
  7. Contextual bandits with general function classes
  8. Related literature
  9. Contextual bandits with general function classes (and finite actions).
  10. Variants of UCB for particular contextual bandit problems.
  11. Contextual bandits with general function classes and infinite actions.
  12. Organization
  13. Upper counterfactual confidence bounds
  14. Introducing UCCB: two equivalent viewpoints
  15. Implicit strategy: maximizing upper confidence bounds in policy space.
  16. ...and 39 more sections

Key Result

Theorem 1

Under Assumption asm realizability and fixing $\delta\in (0,1)$, set the parameter $\beta_t$ in Algorithm alg: uccb to be Then with probability at least $1-\delta$, for all $T\geq 1$, the regret of Algorithm alg: uccb after $T$ rounds is upper bounded by

Theorems & Definitions (26)

  • Theorem 1: Regret for Algorithm \ref{['alg: uccb']}
  • Lemma 1: confidence of policies
  • Lemma 2: contextual potential lemma
  • Lemma 3: the "square trick" relaxation
  • proof
  • Lemma 4: equivalence between Algorithm \ref{['alg: uccb']} and implicit strategy \ref{['eq: policy maximization']}
  • Lemma 5: uniform convergence over all sequences of estimators
  • Corollary 2: extension to infinite $\mathcal{F}$ via parametric dimension
  • Definition 1: covering number
  • Corollary 3: extension to infinite $\mathcal{F}$ via covering number
  • ...and 16 more