Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

THINKSAFE: Self-Generated Safety Alignment for Reasoning Models

Seanie Lee, Sangwoo Park, Yumin Choi, Gyeongman Kim, Minki Kang, Jihun Yun, Dongmin Park, Jongho Park, Sung Ju Hwang

TL;DR

ThinkSafe tackles safety degradation in LRMs caused by post-training RL by enabling self-generated safety alignment through refusal steering. By eliciting harmful-prompt safety reasoning traces directly from the model and training on in-distribution data without external teachers, ThinkSafe minimizes distribution shift while preserving native reasoning. Across Qwen3 and DeepSeek-R1-Distill models, it achieves superior safety while maintaining competitive reasoning, outperforming teacher-distillation baselines and offering substantial efficiency gains over online RL. The approach demonstrates that latent safety knowledge can be unlocked without compromising reasoning, providing a practical, scalable route to robust safety-aligned reasoning models.

Abstract

Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.

THINKSAFE: Self-Generated Safety Alignment for Reasoning Models

TL;DR

ThinkSafe tackles safety degradation in LRMs caused by post-training RL by enabling self-generated safety alignment through refusal steering. By eliciting harmful-prompt safety reasoning traces directly from the model and training on in-distribution data without external teachers, ThinkSafe minimizes distribution shift while preserving native reasoning. Across Qwen3 and DeepSeek-R1-Distill models, it achieves superior safety while maintaining competitive reasoning, outperforming teacher-distillation baselines and offering substantial efficiency gains over online RL. The approach demonstrates that latent safety knowledge can be unlocked without compromising reasoning, providing a practical, scalable route to robust safety-aligned reasoning models.

Abstract

Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.
Paper Structure (48 sections, 5 equations, 13 figures, 7 tables)

This paper contains 48 sections, 5 equations, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Safety risk of LRMs.
  4. Safety-alignment of LRMs.
  5. Self-distillation.
  6. Preliminaries
  7. Problem setup.
  8. Safety fine-tuning.
  9. Our Approach: ThinkSafe
  10. Motivation.
  11. Data generation via refusal steering.
  12. Benign response generation.
  13. Filtering.
  14. Training.
  15. Experiment
  16. ...and 33 more sections

Figures (13)

  • Figure 1: Safety & reasoning performance of the Qwen3 family.
  • Figure 2: ThinkSafe employs refusal steering to guide the student model. This mechanism unlocks the student's latent safety capabilities to generate valid reasoning traces, resulting in responses that are both safe and in-distribution.
  • Figure 3: Comparison of ThinkSafe with online RL; GRPO.
  • Figure 4: Ablation of safety reasoning in R1 model series.
  • Figure 5: Ratio of reasoning gain to safety gain for student models trained on data generated by teachers from the same model family.
  • ...and 8 more figures