SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning

TL;DR

This paper identifies a safety aha moment in large reasoning models (LRMs): the key sentence $K$ following the query understanding $U$ often signals whether the model will respond safely. It introduces SafeKey, a framework with two objectives—Dual-Path Safety Head (DPSH) and Query-Mask Modeling (QMM)—to amplify safety signals in hidden states before $K$ and to compel the model to attend to $U$ when generating $K$. Empirical results across multiple model sizes and jailbreak datasets show SafeKey improves safety generalization to unseen jailbreak prompts and out-of-distribution harmful prompts, while maintaining general abilities. Analyses reveal that SafeKey reshapes internal attention toward $U$ and produces more discriminative hidden representations, enabling safer conditional generation without substantial utility loss. Overall, SafeKey offers a practical, architecture-informed approach to robust safety alignment for LRMs with explicit reasoning stages.

Abstract

Large Reasoning Models (LRMs) introduce a new generation paradigm of explicitly reasoning before answering, leading to remarkable improvements in complex tasks. However, they pose great safety risks against harmful queries and adversarial attacks. While recent mainstream safety efforts on LRMs, supervised fine-tuning (SFT), improve safety performance, we find that SFT-aligned models struggle to generalize to unseen jailbreak prompts. After thorough investigation of LRMs' generation, we identify a safety aha moment that can activate safety reasoning and lead to a safe response. This aha moment typically appears in the `key sentence', which follows models' query understanding process and can indicate whether the model will proceed safely. Based on these insights, we propose SafeKey, including two complementary objectives to better activate the safety aha moment in the key sentence: (1) a Dual-Path Safety Head to enhance the safety signal in the model's internal representations before the key sentence, and (2) a Query-Mask Modeling objective to improve the models' attention on its query understanding, which has important safety hints. Experiments across multiple safety benchmarks demonstrate that our methods significantly improve safety generalization to a wide range of jailbreak attacks and out-of-distribution harmful prompts, lowering the average harmfulness rate by 9.6\%, while maintaining general abilities. Our analysis reveals how SafeKey enhances safety by reshaping internal attention and improving the quality of hidden representations.

Figures (7)

• Figure 1: We find that (1) Supervised fine-tuned LRMs are vulnerable to jailbreaks like multi-turn attacks. (2) The most common thinking pattern of LRMs is to first understand the query, then proceed to think about how to answer. (3) Upper right: Safety aha-moment in the key sentence (K) can lead to a safe response. (4) Bottom right: Based on the query understanding content (U), the SFT model can usually identify unsafe jailbreak queries explicitly, but not when responding to the query. Here, 'A' means the final answer.
• Figure 2: The SafeKey framework: Dual-Path Safety Head contains two safety prediction heads $H_1, H_2$ that take last-layer hidden states on the early generation stage as input and predict the safety of the query. In Query-Mask Modeling, the LRM is trained to predict the key sentence $K$ based on $U$ with query $X$ masked out for attention.
• Figure 3: Ablation on different hidden states used in the Dual-Path Safety Head. The 'U, (X, U)' version, which we used in the end, achieves the best performance.
• Figure 4: Ablation to test the effect of Query-Mask Modeling. QMM has lower harmfulness compared with 'SFT+Key LM', which has the same loss scale.
• Figure 5: Comparison of attention scores between SFT and SafeKey. SafeKey increases the attention between the key sentence $K$ and the query understanding $U$.