Making Harmful Behaviors Unlearnable for Large Language Models

TL;DR

The paper tackles the challenge that fine-tuning LLMs on harmful data can induce undesired behaviors. It introduces security vectors—additional trainable parameters separated from the backbone—that, when activated during fine-tuning, steer the model's outputs toward harmful patterns without updating the core parameters, thereby making the harmful behavior unlearnable; these vectors are deactivated at inference to restore normal behavior. The approach relies on a min-min bi-level optimization to train the vectors on harmful data and then freezes them during downstream fine-tuning, preserving the model's ability to learn other tasks. Empirical results show that security vectors trained on just 100 harmful samples can prevent learning from 1000 harmful examples while maintaining utility on standard benchmarks and enabling safe mixed-task fine-tuning.

Abstract

Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. To meet the requirements of different applications, LLMs are often customized by further fine-tuning. However, the powerful learning ability of LLMs not only enables them to acquire new tasks but also makes them susceptible to learning undesired behaviors. For example, even safety-aligned LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. Can we train LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process. Specifically, we introduce ``security vectors'', a few new parameters that can be separated from the LLM, to ensure LLM's responses are consistent with the harmful behavior. Security vectors are activated during fine-tuning, the consistent behavior makes LLM believe that such behavior has already been learned, there is no need to further optimize for harmful data. During inference, we can deactivate security vectors to restore the LLM's normal behavior. The experimental results show that the security vectors generated by 100 harmful samples are enough to prevent LLM from learning 1000 harmful samples, while preserving the ability to learn other useful information.

Figures (3)

• Figure 1: Illustration of how security vectors make harmful behavior unlearnable. Guided by security vectors, LLM's response is consistent with the harmful response, suggesting that the model does not need to further learn harmful behavior. Concurrently, LLM can still learn from other data because other behaviors are not affected by security vectors.
• Figure 2: An overview of our framework. Given the undesired behavior such as harmful behavior, we first train security vectors on such data, making the harmful behavior "learned" by LLMs. During the fine-tuning phase, security vectors are activated during forward propagation to make LLM's output consistent with harmful responses, thus preventing LLM from learning harmful behaviors. But only LLM's parameters are updated during backward propagation, which allow models to learn from other data. The security vectors can be deactivated during inference, and a clean LLM that has not performed harmful updates can still output benign responses.
• Figure 3: Ablations of epoch and learning rate. All experiments are fine-tuned on ProfileQA+$\text{Harm}_{base}$. We report the results on subsets of RedTeam and LIMA with 20 samples, as well as the full set of ProfileQA. We show the harm score and harm rate on RedTeam, the instruction following rate on LIMA, and EM on ProfileQA.