Non-Stationary Dueling Bandits Under a Weighted Borda Criterion
Joe Suk, Arpit Agarwal
TL;DR
The paper addresses non-stationary dueling bandits with a focus on the Borda winner, introducing a Weighted Borda Score (WBS) framework that unifies Borda and Condorcet objectives. It proves the first adaptive dynamic regret bound for Borda, $\tilde{O}(\tilde{L}^{1/3} K^{1/3} T^{2/3})$, where $\tilde{L}$ counts significant Borda switches, and provides complementary $V_T$-dependent bounds, all without prior knowledge of non-stationarity. Under the General Identifiability Condition (GIC), it also derives adaptive Condorcet bounds in terms of $S_{approx}$ and $V_T$, demonstrating improved rates in regimes with many arms or many spurious winner changes. The algorithms BOSSE and METABOSSE implement the Weighted Borda approach, combining soft-elimination with time-varying exploration and a hierarchical replay mechanism to achieve adaptivity and optimality. Collectively, the work offers a unified, adaptive framework for both Borda and Condorcet dynamic regret and opens avenues for further refinement under weaker identifiability conditions.
Abstract
In $K$-armed dueling bandits, the learner receives preference feedback between arms, and the regret of an arm is defined in terms of its suboptimality to a $\textit{winner}$ arm. The $\textit{non-stationary}$ variant of the problem, motivated by concerns of changing user preferences, has received recent interest (Saha and Gupta, 2022; Buening and Saha, 2023; Suk and Agarwal, 2023). The goal here is to design algorithms with low {\em dynamic regret}, ideally without foreknowledge of the amount of change. The notion of regret here is tied to a notion of winner arm, most typically taken to be a so-called Condorcet winner or a Borda winner. However, the aforementioned results mostly focus on the Condorcet winner. In comparison, the Borda version of this problem has received less attention which is the focus of this work. We establish the first optimal and adaptive dynamic regret upper bound $\tilde{O}(\tilde{L}^{1/3} K^{1/3} T^{2/3} )$, where $\tilde{L}$ is the unknown number of significant Borda winner switches. We also introduce a novel $\textit{weighted Borda score}$ framework which generalizes both the Borda and Condorcet problems. This framework surprisingly allows a Borda-style regret analysis of the Condorcet problem and establishes improved bounds over the theoretical state-of-art in regimes with a large number of arms or many spurious changes in Condorcet winner. Such a generalization was not known and could be of independent interest.