Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward

TL;DR

The paper addresses the vulnerability of language models to short adversarial suffixes by recasting suffix discovery as a reinforcement learning problem with a frozen base model. A suffix policy over discrete tokens is trained with PPO using a calibrated cross-entropy reward that aggregates across label surfaces, supplemented by fluency and KL penalties to stabilize learning. Across five NLP tasks and three diverse models, RL-trained suffixes produce stronger accuracy degradation and richer calibration signals, with notable transferability to unseen models and tasks. The work demonstrates that calibration-aware reinforcement learning provides a robust and lightweight framework for generating transferable adversarial suffixes, offering insights into robustness testing and evaluation of language models.

Abstract

Language models are vulnerable to short adversarial suffixes that can reliably alter predictions. Previous works usually find such suffixes with gradient search or rule-based methods, but these are brittle and often tied to a single task or model. In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization against a frozen model as a reward oracle. Rewards are shaped using calibrated cross-entropy, removing label bias and aggregating across surface forms to improve transferability. The proposed method is evaluated on five diverse NLP benchmark datasets, covering sentiment, natural language inference, paraphrase, and commonsense reasoning, using three distinct language models: Qwen2-1.5B Instruct, TinyLlama-1.1B Chat, and Phi-1.5. Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous adversarial triggers of similar genres.