Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Models

TL;DR

This work tackles the fragility of LLM safety guardrails by proposing GOSV, a global optimization framework that identifies safety-critical attention heads via two activation repatching strategies. It uncovers two spatially distinct safety pathways and a universal threshold of about $0.30$ of heads that, when repatched, cause complete safety breakdown, enabling an inference-time white-box jailbreak. The results show that safety mechanisms are distributed and interdependent, challenging purely local attribution approaches and demonstrating superior attack performance compared to baselines and fine-tuning. Overall, the study advances mechanistic interpretability of LLM safety and emphasizes the need for defenses that account for distributed safety vectors across multiple attention pathways.

Abstract

While Large Language Models (LLMs) are aligned to mitigate risks, their safety guardrails remain fragile against jailbreak attacks. This reveals limited understanding of components governing safety. Existing methods rely on local, greedy attribution that assumes independent component contributions. However, they overlook the cooperative interactions between different components in LLMs, such as attention heads, which jointly contribute to safety mechanisms. We propose \textbf{G}lobal \textbf{O}ptimization for \textbf{S}afety \textbf{V}ector Extraction (GOSV), a framework that identifies safety-critical attention heads through global optimization over all heads simultaneously. We employ two complementary activation repatching strategies: Harmful Patching and Zero Ablation. These strategies identify two spatially distinct sets of safety vectors with consistently low overlap, termed Malicious Injection Vectors and Safety Suppression Vectors, demonstrating that aligned LLMs maintain separate functional pathways for safety purposes. Through systematic analyses, we find that complete safety breakdown occurs when approximately 30\% of total heads are repatched across all models. Building on these insights, we develop a novel inference-time white-box jailbreak method that exploits the identified safety vectors through activation repatching. Our attack substantially outperforms existing white-box attacks across all test models, providing strong evidence for the effectiveness of the proposed GOSV framework on LLM safety interpretability.

Figures (8)

• Figure 1: Overview of our approach: GOSV framework and our attack. Left: The GOSV optimization process employs two activation repatching strategies (Harmful Patching and Zero Ablation) combined with global optimization to identify safety-critical attention heads. Right: Our attack replaces activations at the identified safety-critical heads with the corresponding repatching values at inference time.
• Figure 2: Spatial distribution of safety-critical attention heads identified by GOSV. Top: Head locations identified via Zero Ablation. Bottom: Head locations identified via Harmful Patching. Each cell represents an attention head at layer $l$ and position $h$, with color intensity indicating the selection probability $\sigma(\theta^{(l,h)})$ after optimization.
• Figure 3: Overlap analysis of top-$k$ heads selected by ranking the probabilities $\sigma(\theta^{(l,h)})$ from Harmful Patching and Zero Ablation across different models.
• Figure 4: Progressive intervention analysis across four models showing ASR and perplexity (PPL) as functions of the number of repatched heads for both strategies, revealing that safety mechanisms are encoded in approximately 30% of attention heads across all models.
• Figure 5: Comparison of ASR using different activation repatching strategies across four models. We evaluate Harmful Patching, Zero Ablation, Benign Patching, and Random Patching. Only Harmful Patching and Zero Ablation achieve high ASR, validating that successful attacks require targeting specific safety vectors.