Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Swap Regret Minimization in Combinatorial Bandits

Andreas Kontogiannis, Vasilis Pollatos, Panayotis Mertikopoulos, Ioannis Panageas

TL;DR

This work tackles no-swap regret in adversarial combinatorial bandits where the action set is exponentially large in problem dimensions. The authors introduce a multi-scale, lazy framework (master with ScaleLearners) and instantiate a practical base learner (Lazy-ComBCP) built from barycentric spanners and Carathéodory decomposition to bound external regret; this, in turn, yields a no-swap-regret guarantee with a bound of $\mathcal{O}\left(\frac{T\log(d\log T)}{\log T}\right)$. The results demonstrate tightness in the regime $T\le\exp(\Omega(d^{1/14}))$ and show that the approach achieves polylogarithmic dependence on $N=\mathcal{O}(d^m)$ while maintaining $\mathrm{poly}(d,m)$ per-iteration complexity across many standard combinatorial settings. This advances practical no-swap-regret learning for structured bandits with large action spaces and has potential impact on applications like online shortest paths, spanning trees, and permutation-based tasks by enabling reactive, robust policy updates in adversarial environments.

Abstract

This paper addresses the problem of designing efficient no-swap regret algorithms for combinatorial bandits, where the number of actions $N$ is exponentially large in the dimensionality of the problem. In this setting, designing efficient no-swap regret translates to sublinear -- in horizon $T$ -- swap regret with polylogarithmic dependence on $N$. In contrast to the weaker notion of external regret minimization - a problem which is fairly well understood in the literature - achieving no-swap regret with a polylogarithmic dependence on $N$ has remained elusive in combinatorial bandits. Our paper resolves this challenge, by introducing a no-swap-regret learning algorithm with regret that scales polylogarithmically in $N$ and is tight for the class of combinatorial bandits. To ground our results, we also demonstrate how to implement the proposed algorithm efficiently -- that is, with a per-iteration complexity that also scales polylogarithmically in $N$ -- across a wide range of well-studied applications.

Efficient Swap Regret Minimization in Combinatorial Bandits

TL;DR

This work tackles no-swap regret in adversarial combinatorial bandits where the action set is exponentially large in problem dimensions. The authors introduce a multi-scale, lazy framework (master with ScaleLearners) and instantiate a practical base learner (Lazy-ComBCP) built from barycentric spanners and Carathéodory decomposition to bound external regret; this, in turn, yields a no-swap-regret guarantee with a bound of . The results demonstrate tightness in the regime and show that the approach achieves polylogarithmic dependence on while maintaining per-iteration complexity across many standard combinatorial settings. This advances practical no-swap-regret learning for structured bandits with large action spaces and has potential impact on applications like online shortest paths, spanning trees, and permutation-based tasks by enabling reactive, robust policy updates in adversarial environments.

Abstract

This paper addresses the problem of designing efficient no-swap regret algorithms for combinatorial bandits, where the number of actions is exponentially large in the dimensionality of the problem. In this setting, designing efficient no-swap regret translates to sublinear -- in horizon -- swap regret with polylogarithmic dependence on . In contrast to the weaker notion of external regret minimization - a problem which is fairly well understood in the literature - achieving no-swap regret with a polylogarithmic dependence on has remained elusive in combinatorial bandits. Our paper resolves this challenge, by introducing a no-swap-regret learning algorithm with regret that scales polylogarithmically in and is tight for the class of combinatorial bandits. To ground our results, we also demonstrate how to implement the proposed algorithm efficiently -- that is, with a per-iteration complexity that also scales polylogarithmically in -- across a wide range of well-studied applications.
Paper Structure (29 sections, 28 theorems, 72 equations, 1 figure, 1 table, 2 algorithms)

This paper contains 29 sections, 28 theorems, 72 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background on adversarial combinatorial bandits.
  3. Technical Overview.
  4. Preliminaries
  5. Swap-ComBCP: A No-Swap-Regret Combinatorial Bandit Algorithm
  6. The algorithmic framework
  7. Instantiating Lazy-CombAlg
  8. Proof Sketch of Theorem \ref{['thm:combcp_no_regret_']}
  9. Main Result
  10. Swap Regret Bounds
  11. Per-Iteration Complexity in Well-Studied Combinatorial Settings
  12. Conclusion
  13. Further Related Work
  14. Other Strong Notions of Regret
  15. Further related work on adversarial combinatorial bandits.
  16. ...and 14 more sections

Key Result

Proposition 3.1

For any policy $p\in\Delta(\mathcal{A})$ fixed in an interval $t=1, \ldots, T$, any sequence of rewards $R_t$ and any swap function $\phi: \mathcal{A}\rightarrow \mathcal{A}$ it holds that

Figures (1)

  • Figure 1: Swap Regret Analysis for the Multi-scale Combinatorial Bandit Algorithm for $T=14, K=3$ and $H=3$. Purple lines represent restarts of the ScaleLearners. Green terms correspond to expected cumulative rewards collected by the learners, while red terms correspond to expected cumulative rewards of a swap policy that is optimal in hindsight. In each interval, where a Lazy-CombAlg's policy remains fixed, the optimal swap policy is playing a fixed pure action (see Proposition \ref{['property_laziness']}). Thus, red terms actually correspond to expected cumulative rewards of fixed actions that are optimal in hindsight. Orange terms represent the external regrets of the lazy no-external regret algorithms.

Theorems & Definitions (47)

  • Proposition 3.1: Swap Regret is upper bounded by External Regret for time-invariant policies
  • Lemma 3.2: Swap Regret Decomposition under Bandit Feedback
  • Remark 3.3
  • Remark 3.4
  • Lemma 3.5
  • Remark 3.6: Efficient Sampling
  • Remark 3.7: Mixing step
  • Lemma 3.8
  • Theorem 3.9: No-External-Regret
  • Proposition 3.10: Decomposition of optimal deterministic policy
  • ...and 37 more