Contextual Bandits for Unbounded Context Distributions

TL;DR

The paper tackles nonparametric contextual bandits with unbounded context distributions by deriving minimax lower bounds and proposing two nearest-neighbor UCB strategies. The fixed-k method yields minimax-optimal regret under weak margins and light tails in favorable regimes, while the adaptive-k NN-UCB achieves near-minimax optimal regret across all regimes, matching lower bounds up to log factors and collapsing to bounded-context rates as tail strength grows. The adaptive approach selects the neighborhood size based on local context density and suboptimality gaps, enabling a data-driven bias-variance tradeoff and exploration-exploitation balance. Empirical results on synthesized data and MNIST validate the theoretical benefits, showing substantial improvements over baselines and demonstrating practicality for high-dimensional, heavy-tailed contexts.

Abstract

Nonparametric contextual bandit is an important model of sequential decision making problems. Under $α$-Tsybakov margin condition, existing research has established a regret bound of $\tilde{O}\left(T^{1-\frac{α+1}{d+2}}\right)$ for bounded supports. However, the optimal regret with unbounded contexts has not been analyzed. The challenge of solving contextual bandit problems with unbounded support is to achieve both exploration-exploitation tradeoff and bias-variance tradeoff simultaneously. In this paper, we solve the nonparametric contextual bandit problem with unbounded contexts. We propose two nearest neighbor methods combined with UCB exploration. The first method uses a fixed $k$. Our analysis shows that this method achieves minimax optimal regret under a weak margin condition and relatively light-tailed context distributions. The second method uses adaptive $k$. By a proper data-driven selection of $k$, this method achieves an expected regret of $\tilde{O}\left(T^{1-\frac{(α+1)β}{α+(d+2)β}}+T^{1-β}\right)$, in which $β$ is a parameter describing the tail strength. This bound matches the minimax lower bound up to logarithm factors, indicating that the second method is approximately optimal.

Figures (3)

• Figure 1: Comparison of cumulative regrets of different methods for one dimensional distributions.
• Figure 2: Comparison of cumulative regrets of different methods for one dimensional distributions.
• Figure 3: Cumulative regrets for MNIST dataset.

Theorems & Definitions (41)

• Example 1
• Example 2
• proof
• Theorem 1
• Theorem 2
• proof
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6