Safety Alignment of LMs via Non-cooperative Games

TL;DR

AdvGame develops a non-cooperative Attacker–Defender framework for safety alignment of language models, trained online with pairwise preferences to avoid reward hacking. By jointly optimizing an Attacker and a Defender, the Defender achieves improved safety and maintained utility, while the Attacker evolves into a strong red-teaming agent capable of probing target models. The approach relies on pairwise judges, faithfulness constraints, and KL regularization, and it offers MD/IPO-MD variants that stabilize training through EMA-based off-policy data. Experiments show AdvGame-DPO-MD and AdvGame-IPO-MD provide superior balance across safety, compliance, and utility and exhibit robustness to adaptive attacks, compared to baselines like Self-RedTeam and GRPO. This work suggests game-theoretic, non-zero-sum learning can yield practical improvements for secure and useful LM deployment, with future work extending reward models and attack modalities.

Abstract

Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models.

Figures (6)

• Figure 1: Utility (accuracy) and safety (attack success rate) of different safety post-training methods for Qwen2.5-7B-Instruct. Original---the official instruction-tuned model downloaded from HuggingFace. Self-RedTeam liu2025selfredteam is a self-improvement baseline. Our proposed approach preserves (or improves) the original model's utility while noticeably enhancing its safety against adversarial attacks.
• Figure 2: AdvGame DPO variant overview. An Attacker LM proposes two prompt modifications from a seed prompt, which a Defender LM responds to with the goal of safety. The Attacker and Defender are optimized with preference objectives defined by faithfulness, compliance, and deflection judges.
• Figure 3: Training dynamics of different online RL methods on Qwen2.5-7B. We report train reward for both Attacker and Defender (ranging from -1 to 10). Validation curves are reported on a subset of WJB dataset (256 adversarial harmful and adversarial benign prompt). DPO-MD and IPO-MD show similar behavior, whereas GRPO struggles with high fluctuations and smaller reward.
• Figure 4: Similar to \ref{['f:experiments-dpo-ipo-grpo']} but exploring different setups for AdvGame-DPO-MD on Qwen2.5-7B. Specifically, we study the effect of 1) top -- fixing the Attacker model vs training it along with Defender; 2) middle -- using point-wise judge vs pairwise judge; 3) bottom -- using on-policy model for generation vs EMA model (off-policy). Each row with figures is followed by the corresponding utility/safety evaluations (as a table).
• Figure 5: Similar to \ref{['f:experiments-ablation-curves-main']} but exploring different setups for AdvGame-DPO-MD on Qwen2.5-7B. Specifically, we study the effect of 1) using non-thinking vs thinking chat template for Attacker; 2) using smaller (Qwen2.5-7B) vs bigger (Qwen2.5-32B) model as a judge; 3) using optimistic vs non-optimistic responses for Attacker (see \ref{['sec:method-algo']}). Each row of figures is followed by the corresponding utility/safety evaluations (as a table).