Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention

TL;DR

This paper proposes Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals, highlighting the importance of explicit alignment for reasoning and providing a practical path to safer LRMs.

Abstract

Although Large Reasoning Models (LRMs) have progressed in solving complex problems, their chain-of-thought (CoT) reasoning often contains harmful content that can persist even when the final responses appear safe. We show that this issue still remains in existing methods which overlook the unique significance of safe reasoning, undermining their trustworthiness and posing potential risks in applications if unsafe reasoning is accessible for and exploited by malicious users. We therefore shift our focus to aligning the safety of reasoning itself in this paper and explore process supervision as the solution. However, simply rewarding safe reasoning proves inadequate due to low rollout diversity and limited training signals. To tackle this challenge, we first delve into the characteristics of safe reasoning and uncover several critical insights that 1) safe reasoning is often consolidated by a few critical steps of safety triggers; 2) compliance cues strongly correlate with unsafe continuations; and 3) corrective interventions reliably steer unsafe trajectories towards safer traces. Motivated by these, we propose Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals. Experiments on jailbreak and adversarial safety benchmarks demonstrate that IPO remarkably improves overall safety regarding both reasoning and responses, outperforming SFT-based and RL-based baselines with a relative reduction of over 30% in harmfulness, while preserving excellent performance across diverse reasoning tasks. The results highlight the importance of explicit alignment for reasoning and provide a practical path to safer LRMs.

Figures (14)

• Figure 1: Intervened Preference Optimization (IPO) aligns LRMs towards safe reasoning. We prioritize reasoning safety in LRM alignment, which remains an issue even in recent aligned LRMs, e.g., RealSafe zhang2025realsafe. We propose to intervene in the reasoning of LRMs, correcting unsafe reasoning with safety triggers to safe ones and training on them with preference optimization.
• Figure 1: Reasoning safety trained by GRPO. RL rewarding safe reasoning is relatively effective but insufficient.
• Figure 2: Reasoning and response safety of existing aligned LRMs. We report the harmful ratios of RealSafe and STAR models on three benchmarks in both their reasoning and final responses.
• Figure 3: Distribution of reasoning and response safety in outputs from DS-8B.
• Figure 4: Diversity in safety within a rollout group. Half of harmful prompts for rollout hardly get safe reasoning.