RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards

TL;DR

RSafe tackles the core problem of safeguarding LLMs by introducing a two-stage, reasoning-based guard that uses guided reasoning over safety policies and zero-shot reinforcement learning to align its reasoning with accurate safety judgments. The framework supports dynamic, user-specified safety taxonomies and generates verifiable reasoning traces, enabling robust generalization to unseen or adversarial threats. Empirical results on six safety benchmarks show competitive effectiveness, with strong out-of-distribution robustness, and an adaptive variant that further improves performance under tailored safety requirements. Limitations include reliance on the backbone's reasoning capacity and the need for users to articulate appropriate safety taxonomies, pointing to future work on stronger reasoning backbones and domain-specific safety policies.

Abstract

Large Language Models (LLMs) continue to exhibit vulnerabilities despite deliberate safety alignment efforts, posing significant risks to users and society. To safeguard against the risk of policy-violating content, system-level moderation via external guard models-designed to monitor LLM inputs and outputs and block potentially harmful content-has emerged as a prevalent mitigation strategy. Existing approaches of training guard models rely heavily on extensive human curated datasets and struggle with out-of-distribution threats, such as emerging harmful categories or jailbreak attacks. To address these limitations, we propose RSafe, an adaptive reasoning-based safeguard that conducts guided safety reasoning to provide robust protection within the scope of specified safety policies. RSafe operates in two stages: 1) guided reasoning, where it analyzes safety risks of input content through policy-guided step-by-step reasoning, and 2) reinforced alignment, where rule-based RL optimizes its reasoning paths to align with accurate safety prediction. This two-stage training paradigm enables RSafe to internalize safety principles to generalize safety protection capability over unseen or adversarial safety violation scenarios. During inference, RSafe accepts user-specified safety policies to provide enhanced safeguards tailored to specific safety requirements.

Figures (2)

• Figure 1: An overview of RSafe's training pipeline. RSafe is guided by an instruction prompt that explicitly asks it to reason about the safety of a user input under a set of predefined safety policies. It generates multiple candidate responses in a structured format. These responses are evaluated using a binary rule-based reward function applied to the final prediction: a reward of zero for incorrect predictions and full reward for correct ones. An additional format reward is included to enforce the reasoning schema. The resulting reward signals are used to align RSafe through RL.
• Figure 2: A case study illustrating RSafe's detailed rollout process during training. The full instruction prompt is shown on the left, wrapped with the user prompt $x_i$ and a set of predefined safety policies $\mathcal{S}$. RSafe generates $n$ rollouts as required by the GRPO algorithm; here, we present an example with $n = 3$. The reward function first checks whether the response includes the required <think> tag, and then compares the prediction $y_i$ with the ground-truth label. Only responses that strictly follow the reasoning schema and produce the correct prediction receive the full reward; otherwise, the model receives only the format reward, or zero if the response also fails to follow the required format.