Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay

TL;DR

The paper tackles proactive safety alignment for LLMs by addressing the inefficiencies of static red-teaming and data-intensive training. It introduces Safety Self-Play (SSP), a unified single-model framework in which an LLM simultaneously acts as Attacker and Defender within a reinforcement learning loop, guided by an external safety judge. To prevent forgetting and overfitting, SSP incorporates an Advanced Reflective Experience Replay that stores hard past failures and applies Upper Confidence Bound sampling to replay them strategically, enabling robust adversarial co-evolution. Empirical results across multiple backbones show that SSP reduces jailbreak success rates while preserving core capabilities, outperforming both inference-time and training-time baselines. The work demonstrates a path toward proactive, autonomous safety alignment with practical implications for deploying safer LLMs in dynamic threat environments, while acknowledging computation and data-diversity limitations and outlining avenues for extending to multimodal threats.

Abstract

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to adversarial ``jailbreak'' attacks designed to bypass safety guardrails. Current safety alignment methods depend heavily on static external red teaming, utilizing fixed defense prompts or pre-collected adversarial datasets. This leads to a rigid defense that overfits known patterns and fails to generalize to novel, sophisticated threats. To address this critical limitation, we propose empowering the model to be its own red teamer, capable of achieving autonomous and evolving adversarial attacks. Specifically, we introduce Safety Self- Play (SSP), a system that utilizes a single LLM to act concurrently as both the Attacker (generating jailbreaks) and the Defender (refusing harmful requests) within a unified Reinforcement Learning (RL) loop, dynamically evolving attack strategies to uncover vulnerabilities while simultaneously strengthening defense mechanisms. To ensure the Defender effectively addresses critical safety issues during the self-play, we introduce an advanced Reflective Experience Replay Mechanism, which uses an experience pool accumulated throughout the process. The mechanism employs a Upper Confidence Bound (UCB) sampling strategy to focus on failure cases with low rewards, helping the model learn from past hard mistakes while balancing exploration and exploitation. Extensive experiments demonstrate that our SSP approach autonomously evolves robust defense capabilities, significantly outperforming baselines trained on static adversarial datasets and establishing a new benchmark for proactive safety alignment.

Figures (7)

• Figure 1: Safety Self-Play (SSP) pipeline. A single LLM acts as both attacker and defender. Given a harmful goal, the Attacker generates a jailbreak prompt, which the Defender answers with a defense response. The response is evaluated by a safety judge to produce reward signals. Beyond ongoing self-play, low-reward failure cases are accumulated in an experience pool and selectively revisited using a UCB-based strategy that prioritizes items with low rewards and low sampling frequency.
• Figure 2: A successful attack scenario where the defender fails to identify the malicious intent and generates a detailed response, yielding a high attack reward.
• Figure 3: Reward curves of attacker and defender without an experience pool.
• Figure 4: Reward curves of attacker and defender with an experience pool.
• Figure 5: Evolution of attacker and defender experience pool sizes over training steps.