Understanding and Preserving Safety in Fine-Tuned LLMs

TL;DR

The paper addresses safety-utility drift during fine-tuning of LLMs, where downstream updates can erode safety guardrails.It reveals that safety gradients lie in a low-rank subspace and that safety and utility gradients are negatively correlated, complicating optimization.The authors propose Safety-Preserving Fine-tuning (SPF), which projects out conflicting utility components using a one-shot safety gradient from a single sample, with theoretical guarantees on utility convergence and bounded safety drift.Empirically SPF recovers pre-trained safety alignment and resists deep fine-tuning and jailbreaks across multiple models, offering a practical, efficient defense for Fine-tuning-as-a-Service.

Abstract

Fine-tuning is an essential and pervasive functionality for applying large language models (LLMs) to downstream tasks. However, it has the potential to substantially degrade safety alignment, e.g., by greatly increasing susceptibility to jailbreak attacks, even when the fine-tuning data is entirely harmless. Despite garnering growing attention in defense efforts during the fine-tuning stage, existing methods struggle with a persistent safety-utility dilemma: emphasizing safety compromises task performance, whereas prioritizing utility typically requires deep fine-tuning that inevitably leads to steep safety declination. In this work, we address this dilemma by shedding new light on the geometric interaction between safety- and utility-oriented gradients in safety-aligned LLMs. Through systematic empirical analysis, we uncover three key insights: (I) safety gradients lie in a low-rank subspace, while utility gradients span a broader high-dimensional space; (II) these subspaces are often negatively correlated, causing directional conflicts during fine-tuning; and (III) the dominant safety direction can be efficiently estimated from a single sample. Building upon these novel insights, we propose safety-preserving fine-tuning (SPF), a lightweight approach that explicitly removes gradient components conflicting with the low-rank safety subspace. Theoretically, we show that SPF guarantees utility convergence while bounding safety drift. Empirically, SPF consistently maintains downstream task performance and recovers nearly all pre-trained safety alignment, even under adversarial fine-tuning scenarios. Furthermore, SPF exhibits robust resistance to both deep fine-tuning and dynamic jailbreak attacks. Together, our findings provide new mechanistic understanding and practical guidance toward always-aligned LLM fine-tuning.

Figures (9)

• Figure 1: Illustration of the threat model.
• Figure 2: Utility and safety under fine-tuning across datasets with varying $\alpha$ in Eq.\ref{['eq:obj']}. The optimal $\alpha$ interval is defined as the range where utility gains are significant while safety degradation is controlled.
• Figure 3: Layer-wise singular value spectra of safety, math, and code gradients across various LLMs. The x-axis represents the rank index, and darker shades indicate deeper transformer layers. Safety gradients exhibit sharper spectral decay, implying lower effective dimensionality.
• Figure 4: Layer-wise subspace similarity between single-sample and full-batch safety gradients $\phi(\mathbf{g}^s_\text{single}, \mathbf{g}^s_\text{batch})$ in Llama.
• Figure 5: Step-wise cosine similarity between safety and utility gradients (harm, math, code) in Llama-3.1-8B-Instruct. Negative regions indicate conflict between the safety and task-optimization directions.

Theorems & Definitions (4)

• Lemma 1: Random-subspace norm preservation
• Lemma 2: Projected descent
• Theorem 1: Nonconvex SGD rate with projection
• Theorem 2: Safety-drift bound