What Matters For Safety Alignment?

TL;DR

The paper conducts a large-scale empirical study to identify what matters for safety alignment in LLMs and LRMs, evaluating 32 models across 13 families with six intrinsic characteristics and three external attack modalities. It reveals substantial safety disparities across model families, with reasoning capabilities and post-training practices playing pivotal roles; notably, post-training and knowledge distillation can degrade safety, while MoE-based and reasoning-enabled architectures often fare better against attacks. A striking finding is the outsized effectiveness of response prefix attacks, which dramatically raise ASR across models, underscoring vulnerabilities in text-completion interfaces and user-defined prefixes. The study also highlights that jailbreak methods like roleplay, prompt injection, and GCG remain potent, and that slow thinking can curb some attack types but not all, depending on model and dataset. Collectively, the work argues for explicit safety constraints and objective optimization during training and deployment, along with architectural safeguards and robust monitoring to mitigate evolving threat vectors in LLM/LRM ecosystems.

Abstract

This paper presents a comprehensive empirical study on the safety alignment capabilities. We evaluate what matters for safety alignment in LLMs and LRMs to provide essential insights for developing more secure and reliable AI systems. We systematically investigate and compare the influence of six critical intrinsic model characteristics and three external attack techniques. Our large-scale evaluation is conducted using 32 recent, popular LLMs and LRMs across thirteen distinct model families, spanning a parameter scale from 3B to 235B. The assessment leverages five established safety datasets and probes model vulnerabilities with 56 jailbreak techniques and four CoT attack strategies, resulting in 4.6M API calls. Our key empirical findings are fourfold. First, we identify the LRMs GPT-OSS-20B, Qwen3-Next-80B-A3B-Thinking, and GPT-OSS-120B as the top-three safest models, which substantiates the significant advantage of integrated reasoning and self-reflection mechanisms for robust safety alignment. Second, post-training and knowledge distillation may lead to a systematic degradation of safety alignment. We thus argue that safety must be treated as an explicit constraint or a core optimization objective during these stages, not merely subordinated to the pursuit of general capability. Third, we reveal a pronounced vulnerability: employing a CoT attack via a response prefix can elevate the attack success rate by 3.34x on average and from 0.6% to 96.3% for Seed-OSS-36B-Instruct. This critical finding underscores the safety risks inherent in text-completion interfaces and features that allow user-defined response prefixes in LLM services, highlighting an urgent need for architectural and deployment safeguards. Fourth, roleplay, prompt injection, and gradient-based search for adversarial prompts are the predominant methodologies for eliciting unaligned behaviors in modern models.

Figures (10)

• Figure 1: Comparison of the safety alignment capabilities of the evaluated 32 LLMs/LRMs (scales range 3B$\sim$235B). Attack success rates (ASRs) are averaged over the five AdvBench, XSTest, HarmBench, SorryBench, and StrongReject datasets. Lower ASRs indicate stronger safety alignment.
• Figure 2: Overview of the large-scale benchmark for safety alignment in LLMs and LRMs.
• Figure 3: ASRs on the AdvBench, XSTest, HarmBench, SorryBench, and StrongReject datasets of diverse fast-thinking, instruction-following LLMs with different scales and architectures. Qwen3Guard-Gen-8B is utilized for response safety classification.
• Figure 4: Correlation between ASRs and model scales.
• Figure 5: ASRs on the AdvBench, XSTest, HarmBench, SorryBench, and StrongReject datasets of post-trained LLMs with the same architectures but separate fast and slow thinking modes.