A conversion theorem and minimax optimality for continuum contextual bandits
Arya Akhavan, Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov
TL;DR
This work develops a general static-to-contextual regret conversion for continuum contextual bandits, enabling sub-linear contextual regret from any input non-contextual algorithm via context-space discretization. By imposing a Hölder smoothness in the context with exponent $\gamma$, the authors balance static regret in each context cell against a localization bias, yielding a tunable upper bound that adapts to Lipschitz, convex, and strongly convex/smooth regimes. They construct a concrete algorithm for the strongly convex/smooth case (noisy BCO extension) and, through the general conversion, obtain minimax-optimal contextual regret rates up to logarithmic factors in $T$, with explicit dependence on the context dimension $p$ and action dimension $d$. A matching minimax lower bound shows the necessity of continuity in the contextual dependence for sub-linear contextual regret and confirms near-optimality of the proposed rates. The results provide a unifying, dimension-aware framework for continuum contextual bandits, clarifying when sub-linear contextual regret is achievable and how the rates scale with problem geometry and noise.
Abstract
We study the contextual continuum bandits problem, where the learner sequentially receives a side information vector and has to choose an action in a convex set, minimizing a function associated with the context. The goal is to minimize all the underlying functions for the received contexts, leading to the contextual notion of regret, which is stronger than the standard static regret. Assuming that the objective functions are $γ$-Hölder with respect to the contexts, $0<γ\le 1,$ we demonstrate that any algorithm achieving a sub-linear static regret can be extended to achieve a sub-linear contextual regret. We prove a static-to-contextual regret conversion theorem that provides an upper bound for the contextual regret of the output algorithm as a function of the static regret of the input algorithm. We further study the implications of this general result for three fundamental cases of dependency of the objective function on the action variable: (a) Lipschitz bandits, (b) convex bandits, (c) strongly convex and smooth bandits. For Lipschitz bandits and $γ=1,$ combining our results with the lower bound of Slivkins (2014), we prove that the minimax optimal contextual regret for the noise-free adversarial setting is achieved. Then, we prove that in the presence of noise, the contextual regret rate as a function of the number of queries is the same for convex bandits as it is for strongly convex and smooth bandits. Lastly, we present a minimax lower bound, implying two key facts. First, obtaining a sub-linear contextual regret may be impossible over functions that are not continuous with respect to the context. Second, for convex bandits and strongly convex and smooth bandits, the algorithms that we propose achieve, up to a logarithmic factor, the minimax optimal rate of contextual regret as a function of the number of queries.