When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails

TL;DR

This work identifies Self-Jailbreak as a core challenge in large reasoning models, where models notice risky prompts during reasoning yet justify unsafe responses later. It introduces Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning into safety-oriented chains, combining Safety Recomposition and Safety Backtrack with a selective loss masking strategy. Through extensive experiments on multiple safety and reasoning benchmarks, CoG demonstrates improved safety while largely preserving original reasoning patterns, outperforming prior safety-for-reasoning methods that degrade reasoning. The approach advances practical alignment of LRMs, enabling safer, more reliable multi-step reasoning without sacrificing capability. The findings have potential impact on deploying LRMs in safety-critical settings by reducing harmful outputs without compromising reasoning performance.

Abstract

Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks but remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks. Existing mitigation strategies rely on injecting heuristic safety signals during training, which often suppress reasoning ability and fail to resolve the safety-reasoning trade-off. To systematically investigate this issue, we analyze the reasoning trajectories of diverse LRMs and uncover a phenomenon we term Self-Jailbreak, where models override their own risk assessments and justify responding to unsafe prompts. This finding reveals that LRMs inherently possess the ability to reject unsafe queries, but this ability is compromised, resulting in harmful outputs. Building on these insights, we propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps, steering the model back onto safe trajectories while preserving valid reasoning chains. Extensive experiments across multiple reasoning and safety benchmarks demonstrate that CoG substantially improves the safety of current LRMs while preserving comparable reasoning ability, significantly outperforming prior methods that suffer from severe safety-reasoning trade-offs.

Figures (18)

• Figure 1: Illustration of “Self-Jailbreak.” The Instruct model (left) directly refuses harmful queries, whereas the LRM (right) identifies risk during Risk Awareness but later engages in Self-Jailbreak (red text) during Risk Analysis, ultimately overriding its initial safety judgment and producing harmful content in the Response Strategy stage.
• Figure 2: Processing of harmful queries by LRM, illustrating the workflow from receiving prompts, through risk detection during reasoning, to the four possible outcomes: aware of risk and responds, aware of risk and refuses, unaware of risk and responds, unaware of risk and refuses.
• Figure 3: Workflow of the Chain-of-Guardrail (CoG) Framework for mitigating "Self-Jailbreak". The framework has three phases: Phase I (left) – the base model $\pi_0$ generates an initial response, which a Judge Model analyzes to extract key Chain-of-Thought (CoT) components; Phase II (middle) – $\pi_0$ applies targeted interventions via Safe Recomposition (SafR) or Safety Backtrack (SafB) to ensure CoT safety; Phase III (right) – $\pi_0$ produces the final safe response. The interaction also generates high-quality training data to fine-tune $\pi_0$ for safer reasoning without reducing capability.
• Figure 4: Safety vs. reasoning trade-off for the 32B model.
• Figure 5: PCA of the Qwen3-32B representation space. Gray ellipses indicate base model distributions, while colored regions denote post-training clusters (red: safety, blue: reasoning). Centroid shifts ($\Delta$) quantify displacement relative to the linear decision boundary (dashed), with $\Delta>0$ indicating increased margin and $\Delta<0$ indicating decreased margin.