TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment

TL;DR

This work tackles the challenge of scalable LLM safety alignment by introducing TriPlay-RL, a closed-loop reinforcement learning framework with three roles: an attacker ($M_{ ext{Red}}$), a defender ($M_{ ext{Blue}}$), and an evaluator ($M_{ ext{Eval}}$). Through alternating updates and a tri-level reward system—semantic fidelity, diversity encouragement, and cross-model adversarial objectives—the framework achieves co-evolution of attack strength, safety performance, and evaluation reliability with minimal manual data. Key findings show substantial gains in attack efficacy against diverse defenses, strong safety performance while maintaining general reasoning, and improved stability of safety judgments via multi-expert evaluation. The approach promises a scalable, automated path to robust LLM safety, with practical implications for deployment and ongoing alignment research.

Abstract

In recent years, safety risks associated with large language models have become increasingly prominent, highlighting the urgent need to mitigate the generation of toxic and harmful content. The mainstream paradigm for LLM safety alignment typically adopts a collaborative framework involving three roles: an attacker for adversarial prompt generation, a defender for safety defense, and an evaluator for response assessment. In this paper, we propose a closed-loop reinforcement learning framework called TriPlay-RL that enables iterative and co-improving collaboration among three roles with near-zero manual annotation. Experimental results show that the attacker preserves high output diversity while achieving a 20%-50% improvement in adversarial effectiveness; the defender attains 10%-30% gains in safety performance without degrading general reasoning capability; and the evaluator continuously refines its fine-grained judgment ability through iterations, accurately distinguishing unsafe responses, simple refusals, and useful guidance. Overall, our framework establishes an efficient and scalable paradigm for LLM safety alignment, enabling continuous co-evolution within a unified learning loop.

Figures (7)

• Figure 1: Overview of the proposed tri-role reinforcement learning framework, illustrating the closed-loop interaction among $M_{\mathrm{Red}}$, $M_{\mathrm{Blue}}$, and $M_{\mathrm{Eval}}$.
• Figure 2: The internal mechanism of training loop. $M_{\mathrm{Red}}$ generates adversarial prompts using customized templates to attack $M_{\mathrm{Blue}}$ and other defense models. The reward signal for the $M_{\mathrm{Red}}$ consists of a semantic reward, a diversity penalty, and a weighted score of the $M_{\mathrm{Blue}}$ responses as evaluated by $M_{\mathrm{Eval}}$. Adversarial prompts produced by $M_{\mathrm{Red}}$ are submitted to $M_{\mathrm{Blue}}$, whose outputs are likewise assessed by $M_{\mathrm{Eval}}$, with the evaluation scores serving as the reward signal for training $M_{\mathrm{Blue}}$. The training data for $M_{\mathrm{Eval}}$ consist of adversarial prompts sampled from $P_{\mathrm{Red}}$, the corresponding responses generated by all defense models, and labels determined via multi-expert majority voting.
• Figure 3: ASR of $M_{\mathrm{Red}}$ across different training iterations. It can be observed that ASR steadily improves across the three different defense models. For example, $M_{\mathrm{Red}}$-14B's ASR against DeepSeek-R1-0528-Qwen3-8B increase from 13% to 32%, on Qwen3-8B from 21.84% to 67.75%, and on Llama-3.1-Nemotron-Nano-8B-v1 from 60% to 90%.
• Figure 4: Safety capability evaluation of $M_{\mathrm{Blue}}$ across different training iterations. It shows that although some fluctuations occur during training iterations, the ASR of all three models show a downward trend. Particularly, the ASR of $M_{\mathrm{Blue}}$-14B in the last iteration is the lowest among all models, indicating its great safety capability.
• Figure 5: Accuracy curves of $M_{\mathrm{Eval}}$ on a curated dataset. The accuracy of all three models steadily increases, which in turn yields more accurate and stable reward signals for optimizing both $M_{\mathrm{Red}}$ and $M_{\mathrm{Blue}}$.