Selective Adversarial Attacks on LLM Benchmarks

TL;DR

The paper formalizes selective adversarial attacks on LLM benchmarks, demonstrating that carefully crafted perturbations can reduce a target model's performance while preserving non-targets, thereby challenging fairness and reproducibility in leaderboard-driven evaluation. It presents a TextAttack-based protocol with custom constraints and a surrogate-model framework to generate such perturbations on the MMLU benchmark, and it provides open perturbed datasets for robust evaluation. Key findings show that selectivity exists and can meaningfully alter model rankings, with word-level perturbations (e.g., BAE) typically more impactful than character-level ones, and that surrogate-based paraphrase approaches can amplify selective degradation through iterative training. The work emphasizes the need for perturbation-aware reporting and robustness diagnostics, and it discusses potential defense strategies and risks associated with selective perturbations in benchmark analysis.

Abstract

Benchmarking outcomes increasingly govern trust, selection, and deployment of LLMs, yet these evaluations remain vulnerable to semantically equivalent adversarial perturbations. Prior work on adversarial robustness in NLP has emphasized text attacks that affect many models equally, leaving open the question of whether it is possible to selectively degrade or enhance performance while minimally affecting other models. We formalize this problem and study selective adversarial attacks on MMLU - a widely used benchmark designed to measure a language model's broad general knowledge and reasoning ability across different subjects. Using canonical attacks integrated into TextAttack framework, we introduce a protocol for selectivity assessment, develop a custom constraint to increase selectivity of attacks and propose a surrogate-LLM pipeline that generates selective perturbations. Empirically, we find that selective adversarial attacks exist and can materially alter relative rankings, challenging the fairness, reproducibility, and transparency of leaderboard-driven evaluation. Our results motivate perturbation-aware reporting and robustness diagnostics for LLM evaluation and demonstrate that even subtle edits can shift comparative judgments.

Figures (1)

• Figure 1: Overview of selective adversarial evaluation. (a) LLM benchmarks: start from standard multiple-choice items from MMLU dataset. (b) Constraints: edits are subject to constraints from the categories semantic similarity, grammatical similarity, non-overlapping output, pre-transformation restriction, and a custom constraint for implementing selectivity in the TextAttack framework. (c) Adversarial attacks: using TextAttack, apply character-, word-, and sentence-level transformations to the question or instruction. A surrogate generator proposes paraphrases; candidates are filtered by the constraints and can be improved via SFT/DPO to preferentially degrade a chosen model. (d) Models: three open LLMs (Mistral-7B, Llama-3.1-8B, Qwen-2.5-7B); one is designated as the target and the others as non-targets. (f) Selectivity assessment: compare accuracy on original and perturbed items and categorize outcomes as non-selective (all models change similarly), selective degradation (only the target degrades), or selective improvement (only the target improves).