Follow-the-Perturbed-Leader for Decoupled Bandits: Best-of-Both-Worlds and Practicality

TL;DR

This work addresses decoupled multi-armed bandits, where exploration and exploitation operate on different arms each round. It introduces a Follow-the-Perturbed-Leader policy with Pareto perturbations, achieving Best-of-Both-Worlds guarantees without convex optimization or resampling, thereby delivering minimax optimal regret in adversarial settings and near time-insensitive regret in stochastic (SCA) settings. The method provides a computable exploration distribution and exploits Pareto perturbations to obtain a closed-form-like decision rule, resulting in about 20x faster computation than prior BOBW policies while delivering superior empirical performance. Experiments across adversarial and stochastic regimes demonstrate both regret improvements and practical efficiency, and the work highlights the importance of dedicated decoupled strategies over naive mixtures of exploration and exploitation policies.

Abstract

We study the decoupled multi-armed bandit (MAB) problem, where the learner selects one arm for exploration and one arm for exploitation in each round. The loss of the explored arm is observed but not counted, while the loss of the exploited arm is incurred without being observed. We propose a policy within the Follow-the-Perturbed-Leader (FTPL) framework using Pareto perturbations. Our policy achieves (near-)optimal regret regardless of the environment, i.e., Best-of-Both-Worlds (BOBW): constant regret in the stochastic regime, improving upon the optimal bound of the standard MABs, and minimax optimal regret in the adversarial regime. Moreover, the practicality of our policy stems from avoiding both the convex optimization step required by the previous BOBW policy, Decoupled-Tsallis-INF (Rouyer & Seldin, 2020), and the resampling step that is typically necessary in FTPL. Consequently, it achieves substantial computational improvement, about $20$ times faster than Decoupled-Tsallis-INF, while also demonstrating better empirical performance in both regimes. Finally, we empirically show that our approach outperforms a pure exploration policy, and that naively combining a pure exploration with a standard exploitation policy is suboptimal.

Figures (6)

• Figure 1: Adversarial regret with $\Delta = 0.125$
• Figure 2: Computation time (ms)
• Figure 3: Stochastic regret with $\mathsf{EB}$-$\mathsf{TC}$, a pure exploration policy attaining constant cumulative regret. See Section \ref{['sec: experiment']} for details.
• Figure 4: Stochastic regret of mixed policies, which select the explored arm via $\mathsf{EB}$-$\mathsf{TC}$ and exploited arm according to $\mathsf{FTPL}$ or $\mathsf{FTRL}$.
• Figure 5: Adversarial regret with $\Delta = 0.0625$

Theorems & Definitions (21)

• Theorem 1
• Theorem 2
• Corollary 3
• Lemma 4: Regret decomposition
• Lemma 5
• Lemma 6
• proof
• proof
• proof
• Remark 7