High Probability Bound for Cross-Learning Contextual Bandits with Unknown Context Distributions

TL;DR

This work studies cross-learning contextual bandits with adversarial losses and unknown context distributions, aiming for high-probability regret guarantees. Building on the EXP3-type algorithm of Schneider and Zimmert (2023), it develops a deeper, epoch-aware analysis that exploits weak dependencies across epochs and introduces a surrogate loss sequence to enable tight martingale concentration. The authors prove a high-probability regret bound of $\widetilde{O}(\sqrt{KT\log(1/\delta)})$ for any policy, matching the order of the known expected bound under unknown context distribution. This advances theoretical guarantees for learning in settings like online bidding and sleeping bandits where context distributions are not known in advance and losses can be adversarial.

Abstract

Motivated by applications in online bidding and sleeping bandits, we examine the problem of contextual bandits with cross learning, where the learner observes the loss associated with the action across all possible contexts, not just the current round's context. Our focus is on a setting where losses are chosen adversarially, and contexts are sampled i.i.d. from a specific distribution. This problem was first studied by Balseiro et al. (2019), who proposed an algorithm that achieves near-optimal regret under the assumption that the context distribution is known in advance. However, this assumption is often unrealistic. To address this issue, Schneider and Zimmert (2023) recently proposed a new algorithm that achieves nearly optimal expected regret. It is well-known that expected regret can be significantly weaker than high-probability bounds. In this paper, we present a novel, in-depth analysis of their algorithm and demonstrate that it actually achieves near-optimal regret with high probability. There are steps in the original analysis by Schneider and Zimmert (2023) that lead only to an expected bound by nature. In our analysis, we introduce several new insights. Specifically, we make extensive use of the weak dependency structure between different epochs, which was overlooked in previous analyses. Additionally, standard martingale inequalities are not directly applicable, so we refine martingale inequalities to complete our analysis.

Theorems & Definitions (13)

• Theorem 1: Formal
• Lemma 1: Freedman's Inequality
• Definition 1
• Lemma 2: Lemma 6 and Lemma 7, Zimmert2023
• Lemma 3: Lemma 8, Zimmert2023
• Definition 2
• Lemma 4: Lemma 9, Zimmert2023
• Definition 3
• Lemma 5: Lemma 10, Zimmert2023
• Lemma 6