Truly Adapting to Adversarial Constraints in Constrained MABs

TL;DR

The paper tackles constrained multi-armed bandits under unknown, time-varying constraints in both full and bandit feedback, focusing on non-stationary environments captured by a corruption level $C$. It develops algorithms that achieve sublinear regret and positive constraint violation with rates that degrade gracefully in $C$, without requiring prior knowledge of $C$, by combining optimism with online mirror descent on moving decision spaces and a two-phase exploration approach for bandit losses. The main results include ConOMD-FS for full feedback with $R_T,V_T = \tilde{O}(\sqrt{T}+C)$, ExpOpt-ConOMD for bandit losses achieving $R_T = \tilde{O}(\max\{\sqrt{T}, T^{\beta}\}+C T^{1-\beta})$ and $V_T = \tilde{O}(\sqrt{T}+C)$, and a bandit-constraints variant yielding $R_T = \tilde{O}(\sqrt{T}+C\sqrt{T})$ and $V_T = \tilde{O}(\sqrt{T}+C)$. The work compares favorably with prior best‑of‑both-worlds results and matches stochastic-regime rates in adversarial contexts, while revealing open questions for lower bounds in the bandit case. Overall, it advances the understanding of how to adapt constrained MABs to adversarial/noisy environments with guarantees that scale with constraint non-stationarity.

Abstract

We study the constrained variant of the \emph{multi-armed bandit} (MAB) problem, in which the learner aims not only at minimizing the total loss incurred during the learning dynamic, but also at controlling the violation of multiple \emph{unknown} constraints, under both \emph{full} and \emph{bandit feedback}. We consider a non-stationary environment that subsumes both stochastic and adversarial models and where, at each round, both losses and constraints are drawn from distributions that may change arbitrarily over time. In such a setting, it is provably not possible to guarantee both sublinear regret and sublinear violation. Accordingly, prior work has mainly focused either on settings with stochastic constraints or on relaxing the benchmark with fully adversarial constraints (\emph{e.g.}, via competitive ratios with respect to the optimum). We provide the first algorithms that achieve optimal rates of regret and \emph{positive} constraint violation when the constraints are stochastic while the losses may vary arbitrarily, and that simultaneously yield guarantees that degrade smoothly with the degree of adversariality of the constraints. Specifically, under \emph{full feedback} we propose an algorithm attaining $\widetilde{\mathcal{O}}(\sqrt{T}+C)$ regret and $\widetilde{\mathcal{O}}(\sqrt{T}+C)$ {positive} violation, where $C$ quantifies the amount of non-stationarity in the constraints. We then show how to extend these guarantees when only bandit feedback is available for the losses. Finally, when \emph{bandit feedback} is available for the constraints, we design an algorithm achieving $\widetilde{\mathcal{O}}(\sqrt{T}+C)$ {positive} violation and $\widetilde{\mathcal{O}}(\sqrt{T}+C\sqrt{T})$ regret.

Theorems & Definitions (32)

• Lemma 3.0
• Definition 4.1: $S$-switch dynamic benchmark
• Theorem 4.2
• Theorem 4.3
• Theorem 4.4
• Theorem 5.2
• Theorem 5.3
• Conjecture 6.1
• Lemma B.1
• proof