Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets

TL;DR

Safe Delta addresses the challenge of preserving safety when fine-tuning aligned LLMs on diverse datasets by introducing a Hessian-informed, parameter-level defense. It selects a subset of delta parameters that maximize utility gains under a safety constraint and applies a safety compensation vector inspired by Optimal Brain Surgeon principles to mitigate residual safety loss, with a layer-wise formulation and precomputation of the Hessian inverse. Across extensive experiments on harmful and benign fine-tuning scenarios, Safe Delta consistently improves safety metrics (ASR and Harmfulness Score) while maintaining or enhancing downstream utility, and it generalizes to LoRA and multiple model sizes. The approach offers a practical, scalable defense for real-world fine-tuning services, balancing safety with utility under dataset diversity and offering quantified time costs and limitations for future refinement.

Abstract

Large language models (LLMs) have shown great potential as general-purpose AI assistants across various domains. To fully leverage this potential in specific applications, many companies provide fine-tuning API services, enabling users to upload their own data for LLM customization. However, fine-tuning services introduce a new safety threat: user-uploaded data, whether harmful or benign, can break the model's alignment, leading to unsafe outputs. Moreover, existing defense methods struggle to address the diversity of fine-tuning datasets (e.g., varying sizes, tasks), often sacrificing utility for safety or vice versa. To address this issue, we propose Safe Delta, a safety-aware post-training defense method that adjusts the delta parameters (i.e., the parameter change before and after fine-tuning). Specifically, Safe Delta estimates the safety degradation, selects delta parameters to maximize utility while limiting overall safety loss, and applies a safety compensation vector to mitigate residual safety loss. Through extensive experiments on four diverse datasets with varying settings, our approach consistently preserves safety while ensuring that the utility gain from benign datasets remains unaffected.

Figures (9)

• Figure 1: Existing defense methods struggle when fine-tuning on diverse datasets, causing insufficient protection or utility loss. Left: Data-based methods (BEA, SafeInstr) with 10% augmented safe examples fail to maintain safety as the size of harmful fine-tuning datasets grows. Right: Weight modification methods (Safe LoRA, Resta) fail to balance the utility gained from benign fine-tuning with the need to preserve safety against harmful fine-tuning. ASR represents Attack Success Rate, where a higher value indicates lower safety.
• Figure 2: Preliminary experiments. Guided selection achieves a higher utility score with comparable harmfulness level, compared to random selection. Experiments fine-tune llama-2-7B-Chat on the Dirty Summary dataset. Selective metric is introduced later.
• Figure 3: Overview of Safe Delta. Safe Delta consists of a preparation step performed before fine-tuning and two main steps executed for each fine-tuning request. In the preparation step, the Hessian inverse is computed and saved once, leveraging the original aligned model and safety dataset. For each fine-tuning request, Step 1 selects a subset of delta parameters that maximize total utility improvement while ensuring the safety degradation remains within the specified threshold. Step 2 applies compensatory adjustments to mitigate the safety degradation introduced by the selected delta parameters, ensuring a balance between utility and safety.
• Figure 4: Safety performance as the size of harmful fine-tuning datasets increases. Safe Delta consistently preserves safety.
• Figure 5: Utility-safety trade-off across methods under various hyperparameter settings. Each point represents a method with a specific hyperparameter. (a) Trade-off between benign and harmful fine-tuning. (b) Trade-off within one benign fine-tuning.

Theorems & Definitions (2)

• Theorem 4.1
• Theorem 3.1