Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

TL;DR

This work reveals a surprising vulnerability: fine-tuning on ostensibly benign data can erode LLM safety. It introduces Self-Inf-N, a normalized self-influence-based outlier detector that identifies 100 benign samples most capable of degrading alignment, and demonstrates that fine-tuning on these samples sharply increases harmful outputs across seven mainstream LLMs. The study shows strong cross-architecture transferability and persistence of harm in practical settings like continuous learning and data poisoning, while also examining mitigation strategies that are only partially effective. The findings underscore the need for robust safeguards during benign fine-tuning and motivate further research into defense mechanisms and domain-aware defenses.

Abstract

Recent studies have uncovered a troubling vulnerability in the fine-tuning stage of large language models (LLMs): even fine-tuning on entirely benign datasets can lead to a significant increase in the harmfulness of LLM outputs. Building on this finding, our red teaming study takes this threat one step further by developing a more effective attack. Specifically, we analyze and identify samples within benign datasets that contribute most to safety degradation, then fine-tune LLMs exclusively on these samples. We approach this problem from an outlier detection perspective and propose Self-Inf-N, to detect and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs on 100 outlier samples selected by Self-Inf-N in the benign datasets severely compromises LLM safety alignment. Extensive experiments across seven mainstream LLMs demonstrate that our attack exhibits high transferability across different architectures and remains effective in practical scenarios. Alarmingly, our results indicate that most existing mitigation strategies fail to defend against this attack, underscoring the urgent need for more robust alignment safeguards. Codes are available at https://github.com/GuanZihan/Benign-Samples-Matter.

Figures (13)

• Figure 1: An aligned LLM can reject harmful queries. However, after fine-tuning the LLM on samples filtered from the benign dataset using Self-Inf-N, its safety alignment is easily compromised.
• Figure 2: Fine-tuning Llama-2-7b-chat on the 100 sampled filtered from the Dolly and Alpaca dataset significantly induces harmfulness of LLM's generated content.
• Figure 3: Safety and utility evaluations of LLMs when fine-tuned with benign samples with different token lengths.
• Figure 4: Fine-tuning Llama-2-7b-chat on the 100 sampled filtered from the Dolly dataset with the Self-Inf-N method.
• Figure 5: Transferability of the harmfulness to (a) other architectures and (b) stronger models.