Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bandits in Flux: Adversarial Constraints in Dynamic Environments

Tareq Si Salem

TL;DR

Bandits in Flux tackles adversarial multi-armed bandits under time-varying soft constraints by introducing BCOMD, a primal-dual online mirror-descent algorithm that uses gradient estimators for both costs and constraints. The method leverages an entropic mirror map and a meta-algorithm to adapt to unknown non-stationarity, yielding dynamic regret $\tilde{\mathcal{O}}(\min\{\sqrt{P_T T}, V_T^{1/3} T^{2/3}\})$ and constraint violation $\tilde{\mathcal{O}}(\sqrt{T})$, with empirical demonstrations of state-of-the-art performance. The work unifies online convex optimization, non-stationary bandits, and constrained bandits through a bandit-specific gradient-estimation framework and a dual-update mechanism to enforce long-run feasibility. It also introduces a scalable meta-learning component (MBCOMD) that removes the need for prior knowledge of path-length or temporal variation, enabling robust adaptation in highly dynamic environments. The results have practical implications for systems requiring principled trade-offs between learning efficiency and long-run feasibility under adversarially changing conditions.

Abstract

We investigate the challenging problem of adversarial multi-armed bandits operating under time-varying constraints, a scenario motivated by numerous real-world applications. To address this complex setting, we propose a novel primal-dual algorithm that extends online mirror descent through the incorporation of suitable gradient estimators and effective constraint handling. We provide theoretical guarantees establishing sublinear dynamic regret and sublinear constraint violation for our proposed policy. Our algorithm achieves state-of-the-art performance in terms of both regret and constraint violation. Empirical evaluations demonstrate the superiority of our approach.

Bandits in Flux: Adversarial Constraints in Dynamic Environments

TL;DR

Bandits in Flux tackles adversarial multi-armed bandits under time-varying soft constraints by introducing BCOMD, a primal-dual online mirror-descent algorithm that uses gradient estimators for both costs and constraints. The method leverages an entropic mirror map and a meta-algorithm to adapt to unknown non-stationarity, yielding dynamic regret and constraint violation , with empirical demonstrations of state-of-the-art performance. The work unifies online convex optimization, non-stationary bandits, and constrained bandits through a bandit-specific gradient-estimation framework and a dual-update mechanism to enforce long-run feasibility. It also introduces a scalable meta-learning component (MBCOMD) that removes the need for prior knowledge of path-length or temporal variation, enabling robust adaptation in highly dynamic environments. The results have practical implications for systems requiring principled trade-offs between learning efficiency and long-run feasibility under adversarially changing conditions.

Abstract

We investigate the challenging problem of adversarial multi-armed bandits operating under time-varying constraints, a scenario motivated by numerous real-world applications. To address this complex setting, we propose a novel primal-dual algorithm that extends online mirror descent through the incorporation of suitable gradient estimators and effective constraint handling. We provide theoretical guarantees establishing sublinear dynamic regret and sublinear constraint violation for our proposed policy. Our algorithm achieves state-of-the-art performance in terms of both regret and constraint violation. Empirical evaluations demonstrate the superiority of our approach.
Paper Structure (61 sections, 25 theorems, 127 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 61 sections, 25 theorems, 127 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Online Convex Optimization and Dynamic Comparators.
  4. Non-Stationary MAB.
  5. Non-Stationary MAB with Constraints.
  6. Problem Statement
  7. Policies.
  8. Optimal Comparator Sequence.
  9. Regularity Assumptions.
  10. Incomparability of Regularity Assumptions.
  11. Fundamental Limits.
  12. BCOMD Policy
  13. Gradient Estimators
  14. Mirror Descent Policies
  15. Entropic Instantiation
  16. ...and 46 more sections

Key Result

Proposition 1

(chen2025bridging, besbes2015non) For given $T, n, V_T, P_T$ satisfying $T \ge n \ge 2$ and $V_T \le T/n$, the regret for any policy $\bm{\mathcal{P}}$ is lower bounded as

Figures (2)

  • Figure 1: Non-stationary Trace Generation Setup. The underlying function changes every $2 \times 10^3$ timeslots for six times.
  • Figure 2: Subfigure (a) illustrates the cumulative costs and constraint satisfaction of both BCOMD and R-GP-UCB policies under a non-stationary environment. Subfigure (b) presents the action distribution $\mathcal{A}$ acquired by the BCOMD algorithm.

Theorems & Definitions (49)

  • Proposition 1
  • Definition 1
  • Theorem 1
  • Lemma 1
  • proof : Proof sketch
  • Lemma 2
  • proof : Proof sketch
  • Lemma 3
  • proof : Proof sketch
  • Lemma 4
  • ...and 39 more