Efficient Adversarial Attacks on High-dimensional Offline Bandits

TL;DR

A novel threat model is introduced in which an attacker exploits offline data in high-dimensional settings to hijack the bandit's behavior, and it is proved that as input dimensionality increases, the perturbation norm required for a successful attack decreases, making modern applications such as image evaluation especially vulnerable.

Abstract

Bandit algorithms have recently emerged as a powerful tool for evaluating machine learning models, including generative image models and large language models, by efficiently identifying top-performing candidates without exhaustive comparisons. These methods typically rely on a reward model, often distributed with public weights on platforms such as Hugging Face, to provide feedback to the bandit. While online evaluation is expensive and requires repeated trials, offline evaluation with logged data has become an attractive alternative. However, the adversarial robustness of offline bandit evaluation remains largely unexplored, particularly when an attacker perturbs the reward model (rather than the training data) prior to bandit training. In this work, we fill this gap by investigating, both theoretically and empirically, the vulnerability of offline bandit training to adversarial manipulations of the reward model. We introduce a novel threat model in which an attacker exploits offline data in high-dimensional settings to hijack the bandit's behavior. Starting with linear reward functions and extending to nonlinear models such as ReLU neural networks, we study attacks on two Hugging Face evaluators used for generative model assessment: one measuring aesthetic quality and the other assessing compositional alignment. Our results show that even small, imperceptible perturbations to the reward model's weights can drastically alter the bandit's behavior. From a theoretical perspective, we prove a striking high-dimensional effect: as input dimensionality increases, the perturbation norm required for a successful attack decreases, making modern applications such as image evaluation especially vulnerable. Extensive experiments confirm that naive random perturbations are ineffective, whereas carefully targeted perturbations achieve near-perfect attack success rates ...

Figures (11)

• Figure 1: Comparison of attack performance on linear and nonlinear reward models using OSA, Trajectory-Free, and Full Trajectory methods. All attacks achieve a 100% success rate. The attack times (in seconds) are displayed on a logarithmic scale in the plot, highlighting the superior efficiency of OSA and illustrating the effect of varying parameters $K$ and $d$ (or $\max_i W_i$) on attack duration.
• Figure 2: Left: Effect of increasing input dimensionality on attack magnitude. Both the $\ell_2$ and $\ell_\infty$ norms of the perturbation decrease as the input dimensionality increases, indicating that higher-dimensional inputs are more vulnerable to attacks. Experimental settings: $T=100$, $K=3$, ASR$=100\%$. Right: Effect of applying the proposed defense on attack success rates. Shuffling a portion of the logged data before running the bandit algorithm significantly reduces the effectiveness of attacks. Specifically, we shuffle the first $T/2$ of the logged data before execution.
• Figure 3: Distribution of ASR values for the Aesthetic and Image reward models under the OSA method across two configurations. Each subplot shows the ASR distribution for one reward model.
• Figure 4: Left: Attack success rate as a function of the hidden layer width in the reward model ($T=100$). The results show that a sufficiently wide hidden layer is required for the attack to succeed; once the width exceeds 750 neurons, the ASR reaches 100%. Right: Attack success rate versus the $\ell_2$ norm of random noise perturbations. The results show that even with increasing perturbation magnitude, the ASR remains nearly constant, demonstrating that random noise is largely ineffective in attacking the bandit algorithm.
• Figure 5: Our pre-training attack successfully hijacks the Fast–Slow robust bandit robust_sab1, achieving 100% success with minimal $\ell_2$-norm perturbation ($\approx 0.3$) for $K \in \{3,5\}$, $T = 1000$, and $d = 1000$.

Theorems & Definitions (12)

• Theorem 3.1: Linear Reward Model
• Corollary 3.2: Overparameterized Neural Networks
• Theorem 3.3: Feasibility Guarantee
• Theorem 3.4: $\ell_2$-norm of High-dimensional Attack
• proof : Proof of Theorem \ref{['thm:optLinearQP']}
• proof : Proof of Theorem \ref{['thm:feasibility']}
• proof : Proof of Theorem \ref{['thm:attackSizeGuarantee']}
• Lemma B.1
• proof
• Lemma B.2