Adversarial Combinatorial Bandits with Switching Costs

TL;DR

This work analyzes adversarial combinatorial bandits with a per-arm switching cost $\\lambda$ under bandit and semi-bandit feedback. It derives minimax lower bounds via Yao's principle using specialized stochastic loss sequences and provides batched algorithms—Batched-Exp2 with John's exploration for bandit feedback and Batched-BROAD for semi-bandit feedback—that nearly match these bounds up to polylog factors. The bandit-bound is $\\tilde{O}( (\\lambda K)^{1/3} (T I)^{2/3} )$ with an accompanying $I$-dependent term in the upper bound, while the semi-bandit-bound is $\\tilde{O}( (\\lambda K)^{1/3} (T I)^{2/3} )$; both demonstrate the trade-off between switching costs, batch size, and combinatorial complexity. Numerical results corroborate the theoretical results, showing significant regret reductions of the proposed batched methods compared to baselines across CIN and CDN adversaries, thereby illustrating practical effectiveness in dynamic reconfiguration settings with switching costs.

Abstract

We study the problem of adversarial combinatorial bandit with a switching cost $λ$ for a switch of each selected arm in each round, considering both the bandit feedback and semi-bandit feedback settings. In the oblivious adversarial case with $K$ base arms and time horizon $T$, we derive lower bounds for the minimax regret and design algorithms to approach them. To prove these lower bounds, we design stochastic loss sequences for both feedback settings, building on an idea from previous work in Dekel et al. (2014). The lower bound for bandit feedback is $ \tildeΩ\big( (λK)^{\frac{1}{3}} (TI)^{\frac{2}{3}}\big)$ while that for semi-bandit feedback is $ \tildeΩ\big( (λK I)^{\frac{1}{3}} T^{\frac{2}{3}}\big)$ where $I$ is the number of base arms in the combinatorial arm played in each round. To approach these lower bounds, we design algorithms that operate in batches by dividing the time horizon into batches to restrict the number of switches between actions. For the bandit feedback setting, where only the total loss of the combinatorial arm is observed, we introduce the Batched-Exp2 algorithm which achieves a regret upper bound of $\tilde{O}\big((λK)^{\frac{1}{3}}T^{\frac{2}{3}}I^{\frac{4}{3}}\big)$ as $T$ tends to infinity. In the semi-bandit feedback setting, where all losses for the combinatorial arm are observed, we propose the Batched-BROAD algorithm which achieves a regret upper bound of $\tilde{O}\big( (λK)^{\frac{1}{3}} (TI)^{\frac{2}{3}}\big)$.

Figures (5)

• Figure 1: An illustration of the definition of parent time $\rho(t)$. There is an arrow from each time $t$-th parent time $\rho(t)$ to $t$. For example, $\rho(3)=2$ and $\rho(4)=0$.
• Figure 2: An illustration of the definition of $W_t^i$ for $i\in [K]$. The value of $W_t^i$ is obtained by summing the i.i.d. Gaussian variables $\xi_{t'}^i$'s on the edges along the path from $W_0^i,$ i.e. summing over all $t'\in \mathcal{S}(t) \cup \{t\} \setminus \{0\}$. For example, $W_7^i=$$\xi_4^i +\xi_6^i+\xi_7^i$.
• Figure 3: An illustration of the batches in algorithm. The whole time horizon are divided into batches and during each batch, the player does not change the choice of the combinatorial arm.
• Figure 4: Comparison of the performance of different algorithms under Bandit Feedback
• Figure 5: Comparison of the performances of different algorithms under Semi-bandit Feedback

Theorems & Definitions (19)

• Theorem 1
• Theorem 2
• Theorem 3: Informal
• Theorem 4: Informal
• Lemma 5
• Lemma 6
• Lemma 7
• Lemma 8
• Theorem 9
• Theorem 10