Large Reasoning Models Learn Better Alignment from Flawed Thinking

TL;DR

The paper addresses the brittleness of large reasoning models in safety alignment when their chain-of-thought is seeded with flawed premises. It introduces RECAP, a post-training RLHF method that trains on a mix of counter-aligned cot prefills and standard prompts, teaching models to override unsafe or overly conservative reasoning to produce safe, helpful outputs without extra inference cost. Empirical results show substantial gains in safety and jailbreak robustness, reductions in overrefusal, and small improvements in math reasoning, with maintained token budgets and increased self-reflection. RECAP also demonstrates robustness to adaptive attacks, and ablations reveal how prefilling ratio, length, and source shape performance, underscoring its practical potential for robust, scalable alignment in LRMs.

Abstract

Large reasoning models (LRMs) "think" by generating structured chain-of-thought (CoT) before producing a final answer, yet they still lack the ability to reason critically about safety alignment and are easily biased when a flawed premise is injected into their thought process. We propose RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a principled reinforcement learning (RL) method for post-training that explicitly teaches models to override flawed reasoning trajectories and reroute to safe and helpful responses. RECAP trains on a mixture of synthetically generated counter-aligned CoT prefills and standard prompts, requires no additional training cost or modifications beyond vanilla reinforcement learning from human feedback (RLHF), and substantially improves safety and jailbreak robustness, reduces overrefusal, and preserves core reasoning capability -- all while maintaining inference token budget. Extensive analysis shows that RECAP-trained models engage in self-reflection more frequently and remain robust under adaptive attacks, preserving safety even after repeated attempts to override their reasoning.

Figures (4)

• Figure 1: RECAP trains lrm on a mixture of counter-aligned prefilled and standard prompts. Harmful prompts are prefilled with unsafe reasoning, and benign prompts with refusal reasoning, forcing the model to override flawed trajectories to achieve high rewards. This simple recipe teaches models to internalize safety values and remain robust under both clean and adversarial reasoning traces, with no extra cost beyond standard rlhf.
• Figure 2: Average number of tokens generated at inference for DSQwen-14B under RECAP vs.dapo across safety, overrefusal, and math benchmarks. RECAP maintains a comparable total token budget to dapo.
• Figure 3: Three key factors drive the effectiveness of RECAP: (a) the prefilling ratio and (b) the prefilling length govern the trade-off between safety and overrefusal, while (c) the prefilling source must be counter-aligned rather than aligned. All experiments are conducted on DSLlama-8B with safety and overrefusal prompts, applying cot prefilling only to the safety subset.
• Figure 4: RECAP-trained lrm engage in self-reflection far-more ofthen than vanilla rlhf, frequently revising unsafe or mistaken reasoning mid-trajectory.

Theorems & Definitions (3)

• Theorem 1
• Definition 1: Clipped surrogate
• proof