Unifying Adversarial Robustness and Training Across Text Scoring Models

TL;DR

The paper tackles the fragmented study of adversarial robustness in language models by unifying robustness goals across text scoring models—dense retrievers, rerankers, and reward models—through a text-scoring lens. It introduces multiple adversarial training signals (Rudimentary, HotFlip, PGD, Content Injection, Paraphrasing) and a Combined training strategy, demonstrating that robustness can transfer across threat classes and model roles without sacrificing effectiveness. The study applies these methods to RLHF, showing adversarially trained reward models mitigate reward hacking and yield better-aligned policies, supported by extensive evaluation across retrieval, ranking, and alignment benchmarks. The findings advocate for a cross-cutting defense framework, provide actionable training recipes, and underscore the practical value of robustness in safer, more reliable language-model systems.

Abstract

Research on adversarial robustness in language models is currently fragmented across applications and attacks, obscuring shared vulnerabilities. In this work, we propose unifying the study of adversarial robustness in text scoring models spanning dense retrievers, rerankers, and reward models. This motivates adapting both attacks and adversarial training methods across model roles. Unlike open-ended generation, text scoring failures are directly testable: an attack succeeds when an irrelevant or rejected text outscores a relevant or chosen one. Using this principled lens of text scoring, we demonstrate that current adversarial training formulations for language models are often short-sighted, failing to effectively generalize across attacks. To address this, we introduce multiple adversarial training methods for text scoring models and show that combining complementary training methods can yield strong robustness while also improving task effectiveness. We also highlight the practical value of our approach for RLHF, showing that our adversarially trained reward models mitigate reward hacking and support the training of better-aligned LLMs. We provide our code and models for further study.

Figures (12)

• Figure 1: Average reward and KL per token over training batches during RLHF using base reward models and adversarially trained combined reward models at medium and high training strengths. We test KL penalty coefficients $\beta$ of 0.01 and 0.02. An exponential moving average is plotted to smooth the data.
• Figure 2: Examples of adversarial manipulations across language model tasks and types. Adversarial training studies should not be fragmented by application because retrievers, rerankers, reward models, and generative LLMs all share similar vulnerabilities. We include complete examples below.
• Figure 3: Adversarial manipulations against a BERT-base dense retriever, showing all four attack categories studied (rudimentary manipulations, HotFlip-guided token swaps, MLM-guided word/token swaps, and content injection).
• Figure 4: Adversarial manipulations against a Qwen3-0.6B reranker, showing all four attack categories studied (rudimentary manipulations, HotFlip-guided token swaps, MLM-guided word/token swaps, and content injection).
• Figure 5: Adversarial manipulations against a Llama-3.2-3B-Instruct reward model, showing all four attack categories studied (rudimentary manipulations, HotFlip-guided token swaps, MLM-guided word/token swaps, and content injection).