Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Batched Nonparametric Contextual Bandits

Rong Jiang, Cong Ma

TL;DR

This work studies batched nonparametric contextual bandits where rewards depend smoothly on covariates and policy updates occur only at batch ends. The authors derive a minimax regret lower bound under batch constraints and introduce BaSEDB, a Batched Successive Elimination with Dynamic Binning algorithm, which achieves the optimal regret up to logarithmic factors by adaptively partitioning the covariate space into bins whose widths align with batch sizes. They prove that static binning is suboptimal in the batched setting and show that a nearly constant number of policy updates, $M = O( ext{log} ext{log} T)$, suffices to match the fully online rate, while recognizing a polynomial penalty if the unknown margin parameter $oldsymbol{oldsymbol{oldsymbol{oldsymbol{b}}}}$ is not known. The work further demonstrates the necessity of dynamic binning, provides a tree-based interpretation of the algorithm, and discusses extensions, adaptivity to $oldsymbol{oldsymbol{oldsymbol{oldsymbol{b}}}}$, and potential improvements in logarithmic factors.

Abstract

We study nonparametric contextual bandits under batch constraints, where the expected reward for each action is modeled as a smooth function of covariates, and the policy updates are made at the end of each batch of observations. We establish a minimax regret lower bound for this setting and propose a novel batch learning algorithm that achieves the optimal regret (up to logarithmic factors). In essence, our procedure dynamically splits the covariate space into smaller bins, carefully aligning their widths with the batch size. Our theoretical results suggest that for nonparametric contextual bandits, a nearly constant number of policy updates can attain optimal regret in the fully online setting.

Batched Nonparametric Contextual Bandits

TL;DR

This work studies batched nonparametric contextual bandits where rewards depend smoothly on covariates and policy updates occur only at batch ends. The authors derive a minimax regret lower bound under batch constraints and introduce BaSEDB, a Batched Successive Elimination with Dynamic Binning algorithm, which achieves the optimal regret up to logarithmic factors by adaptively partitioning the covariate space into bins whose widths align with batch sizes. They prove that static binning is suboptimal in the batched setting and show that a nearly constant number of policy updates, , suffices to match the fully online rate, while recognizing a polynomial penalty if the unknown margin parameter is not known. The work further demonstrates the necessity of dynamic binning, provides a tree-based interpretation of the algorithm, and discusses extensions, adaptivity to , and potential improvements in logarithmic factors.

Abstract

We study nonparametric contextual bandits under batch constraints, where the expected reward for each action is modeled as a smooth function of covariates, and the policy updates are made at the end of each batch of observations. We establish a minimax regret lower bound for this setting and propose a novel batch learning algorithm that achieves the optimal regret (up to logarithmic factors). In essence, our procedure dynamically splits the covariate space into smaller bins, carefully aligning their widths with the batch size. Our theoretical results suggest that for nonparametric contextual bandits, a nearly constant number of policy updates can attain optimal regret in the fully online setting.
Paper Structure (63 sections, 15 theorems, 118 equations, 4 figures, 2 algorithms)

This paper contains 63 sections, 15 theorems, 118 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Main contributions
  3. Related work
  4. Nonparametric contextual bandits.
  5. Batch learning.
  6. Switching cost.
  7. Problem setup
  8. Assumptions
  9. Fundamental limits of batched nonparametric bandits
  10. Proof of Theorem \ref{['thm:lower-bound']}
  11. Step 1: Relating regret to inferior sampling rate.
  12. Step 2: Introducing the family of reward instances for $t_{i}$.
  13. Step 3: Lower bounding the inferior sampling rate.
  14. Step 4: Combining bounds together.
  15. Lower bound for general $M$-batch policy
  16. ...and 48 more sections

Key Result

Theorem 1

Suppose that $\alpha\beta\le1$, and assume that $P_{X}$ is the uniform distribution on $\mathcal{X}=[0,1]^{d}$. For any $M$-batch policy $(\Gamma,\pi)$ where $\Gamma$ is prespecified, there exists a nonparametric bandit instance in $\mathcal{F}(\alpha,\beta)$ such that the regret of $(\Gamma,\pi)$ o where $\tilde{D}>0$ is a constant independent of $T$ and $M$.

Figures (4)

  • Figure 1: An example of the tree growing process for $d=1, M=3, G=\{4,3,1\}$. The root node is at depth 0. For the first batch, the 4 nodes located at depth 1 of the tree were used. Both $[\frac{1}{4},\frac{1}{2})$ and $[\frac{1}{2},\frac{3}{4})$ only had one active arm remaining so they were not further split and remained in the set of active nodes (green). Meanwhile, $|\mathcal{I}_{[0,\frac{1}{4})}|=|\mathcal{I}_{[\frac{3}{4},1]}|=2$ so each of them was split into 3 smaller nodes, and both nodes were marked as inactive (red). For the second batch, all the green nodes were actively used but arm elimination was performed at the end of batch 2 only for nodes located at depth 2 (the green nodes at depth 1 already have 1 active arm remaining so there is no need to eliminate again).
  • Figure 2: Regret vs. batch budget $M$.
  • Figure 3: Instance with $g>z$. Each bin $B$ produced by $\hat{\pi}_{\mathrm{static}}$ has width $1/g$.
  • Figure 4: Instance with $g<z$. Each bin $B$ produced by $\hat{\pi}_{\mathrm{static}}$ has width $1/g$.

Theorems & Definitions (22)

  • Remark 1
  • Theorem 1
  • Lemma 1: Lemma 3.1 in rigollet2010nonparametric
  • Theorem 2
  • Theorem 3
  • Corollary 1
  • Theorem 4
  • Theorem 5
  • Lemma 2
  • proof
  • ...and 12 more