Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback

TL;DR

The paper tackles contextual contextual dueling bandits with adversarial feedback, where a strong adversary may flip preference labels. It introduces RCDB, an uncertainty-weighted maximum-likelihood estimator-based algorithm, achieving a near-minimax regret of $\widetilde{O}(d\sqrt{T}/\kappa + dC/\kappa)$ under known adversarial budget and near-optimal lower bounds. For sigmoid links, it further refines the approach to RCDB-S by leveraging local derivatives, yielding a leading-term regret of $\widetilde{O}(d B^{1.5}\sqrt{T} + d B C/\kappa)$ with reduced $\kappa$-dependence. The work demonstrates robustness against adversarial feedback through theoretical guarantees and empirical validation against state-of-the-art dueling bandit methods, contributing a principled framework for secure preference-based learning in contextual settings.

Abstract

Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM). However, the effectiveness of this approach can be influenced by adversaries, who may intentionally provide misleading preferences to manipulate the output in an undesirable or harmful direction. To tackle this challenge, we study a specific model within this problem domain--contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary. We propose an algorithm, namely robust contextual dueling bandits, which is based on uncertainty-weighted maximum likelihood estimation. Our algorithm achieves an $\tilde O(d\sqrt{T}/κ+dC/κ)$ regret bound, where $T$ is the number of rounds, $d$ is the dimension of the context, $κ$ is the lower bound of the derivative of the link function, and $ 0 \le C \le T$ is the total number of adversarial feedback. We also prove a lower bound to show that our regret bound is nearly optimal, both in scenarios with and without ($C=0$) adversarial feedback. Our work is the first to achieve nearly minimax optimal regret for dueling bandits in the presence of adversarial preference feedback. Additionally, for the sigmoid link function, we develop a novel algorithm that takes into account the effect of local derivatives in maximum likelihood estimation (MLE) analysis through a refined method for estimating the link function's derivative. This method helps us to eliminate the $κ$ dependence in the leading term with respect to $T$, which reduces the exponential dependence on the parameter radius $B$ to a polynomial dependence. We conduct experiments to evaluate our proposed algorithm against various types of adversarial feedback. Experimental results demonstrate its superiority over the state-of-the-art dueling bandit algorithms in the presence of adversarial feedback.

Figures (2)

• Figure 1: Comparison of RCDB (Our Algorithm \ref{['alg:main']}), MaxInpsaha2021optimal, CoLSTIMbengs2022stochastic and MaxPairUCBdi2023variance. We report the cumulative regret with various adversarial attack methods (Greedy, Random, Adversarial, Misleading). For the baselines, the parameters are carefully tuned to achieve better results with different attack methods. The total number of adversarial feedback is $C=\lceil\sqrt{T}\rceil$.
• Figure 2: The relationship between cumulative regret and the number of adversarial feedback $C$. For this specific experiment, we employ the "greedy attack" method to generate the adversarial feedback. $C$ is selected from the set $[20,40,60,80,100,120,140,160,180,200]$ (10 adversarial levels).

Theorems & Definitions (17)

• Remark 3.3
• Remark 4.1
• Lemma 5.1
• Remark 5.2
• Theorem 5.3
• Theorem 5.4
• Theorem 5.5
• Remark 5.6
• Theorem 6.1
• Remark 6.2