Fusing Reward and Dueling Feedback in Stochastic Bandits

TL;DR

This work introduces the dueling-reward multi-armed bandit (DR-MAB) problem, where each decision yields both an absolute reward and a relative dueling outcome. It establishes a regret lower bound that shows an optimal fusion need only pay the smaller of the reward-based or dueling-based costs for each suboptimal arm, and it proposes two fusion strategies. ElimFusion fuses information through a shared candidate arm set but incurs a suboptimal $K$-dependent term; DecoFusion decomposes arms and uses randomized decision making to achieve regret matching the lower bound up to a constant, with constant regret when $\alpha$ is at the extremes $0$ or $1$. Experiments confirm the theoretical results, showing DecoFusion outperforming baselines and validating the constant-regret phenomenon under extreme fusion weights.

Abstract

This paper investigates the fusion of absolute (reward) and relative (dueling) feedback in stochastic bandits, where both feedback types are gathered in each decision round. We derive a regret lower bound, demonstrating that an efficient algorithm may incur only the smaller among the reward and dueling-based regret for each individual arm. We propose two fusion approaches: (1) a simple elimination fusion algorithm that leverages both feedback types to explore all arms and unifies collected information by sharing a common candidate arm set, and (2) a decomposition fusion algorithm that selects the more effective feedback to explore the corresponding arms and randomly assigns one feedback type for exploration and the other for exploitation in each round. The elimination fusion experiences a suboptimal multiplicative term of the number of arms in regret due to the intrinsic suboptimality of dueling elimination. In contrast, the decomposition fusion achieves regret matching the lower bound up to a constant under a common assumption. Extensive experiments confirm the efficacy of our algorithms and theoretical results.

Figures (3)

• Figure 1: Regret bounds for DR-MAB
• Figure 2: Regret comparison in different settings
• Figure 3: Regret comparison in different settings (continue)

Theorems & Definitions (10)

• Definition 2.1: Consistent algorithm
• Lemma 2.2
• Theorem 2.3
• Corollary 2.4
• Theorem 3.1
• Theorem 4.1: Regret upper bound of Algorithm \ref{['alg:defusion']}
• proof : Proof of Lemma \ref{['lma:information-lower-bound']}
• proof : Proof of Theorem \ref{['thm:regret-lower-bound']} (regret lower bound)
• proof : Proof of Theorem \ref{['thm:elimination-fusion']}
• proof : Proof of Regret Upper Bound of Algorithm \ref{['alg:defusion']}