Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Immunization against harmful fine-tuning attacks

Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, Jan Batzner, Hassan Sajjad, Frank Rudzicz

TL;DR

The paper formalizes the risk of harmful fine-tuning attacks (HFTA) on safety-aligned LLMs and proposes Immunization, a framework of four conditions (Resistance, Stability, Generalization, Trainability) evaluated within a training-budget threat model. It provides concrete guidelines to rigorously assess defenses, including theoretical guarantees, dataset handling, adaptive-attack accounting, and cross-domain generalization. A demonstration illustrates how an adversarial objective can be used to immunize a model against harmful training, while acknowledging limitations like scope to supervised fine-tuning and the broader harm-definition debate. The work aims to standardize defense research in this emerging area and promote defense-in-depth for safer model deployment.

Abstract

Large Language Models (LLMs) are often trained with safety guards intended to prevent harmful text generation. However, such safety training can be removed by fine-tuning the LLM on harmful datasets. While this emerging threat (harmful fine-tuning attacks) has been characterized by previous work, there is little understanding of how we should proceed in constructing and validating defenses against these attacks especially in the case where defenders would not have control of the fine-tuning process. We introduce a formal framework based on the training budget of an attacker which we call "Immunization" conditions. Using a formal characterisation of the harmful fine-tuning problem, we provide a thorough description of what a successful defense must comprise of and establish a set of guidelines on how rigorous defense research that gives us confidence should proceed.

Immunization against harmful fine-tuning attacks

TL;DR

The paper formalizes the risk of harmful fine-tuning attacks (HFTA) on safety-aligned LLMs and proposes Immunization, a framework of four conditions (Resistance, Stability, Generalization, Trainability) evaluated within a training-budget threat model. It provides concrete guidelines to rigorously assess defenses, including theoretical guarantees, dataset handling, adaptive-attack accounting, and cross-domain generalization. A demonstration illustrates how an adversarial objective can be used to immunize a model against harmful training, while acknowledging limitations like scope to supervised fine-tuning and the broader harm-definition debate. The work aims to standardize defense research in this emerging area and promote defense-in-depth for safer model deployment.

Abstract

Large Language Models (LLMs) are often trained with safety guards intended to prevent harmful text generation. However, such safety training can be removed by fine-tuning the LLM on harmful datasets. While this emerging threat (harmful fine-tuning attacks) has been characterized by previous work, there is little understanding of how we should proceed in constructing and validating defenses against these attacks especially in the case where defenders would not have control of the fine-tuning process. We introduce a formal framework based on the training budget of an attacker which we call "Immunization" conditions. Using a formal characterisation of the harmful fine-tuning problem, we provide a thorough description of what a successful defense must comprise of and establish a set of guidelines on how rigorous defense research that gives us confidence should proceed.
Paper Structure (26 sections, 6 equations, 2 figures, 2 tables)

This paper contains 26 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Motivation
  2. Threat Model for HFTAs
  3. Goal
  4. Strategy
  5. Capabilities
  6. Immunization Conditions
  7. Guidelines For Future Research
  8. Establishing resistance
  9. Harmful dataset selection and creation
  10. Measuring resistance
  11. Ensuring Stability
  12. Adaptive Attacks
  13. Encouraging Generalization
  14. Evaluating Trainability
  15. Conclusion
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Harmful fine-tuning attacks train safety-aligned models for harmful purposes after they are released. Backdoor attacks occur before model release and involve stealthiness (hiding a trigger). Adversarial attacks occur after release but at inference-time.
  • Figure 2: An analysis of the harmful training attack on the initial model (min_harmful_loss) and the immunized model (min_adversarial_loss). Our immunization method prevents harmful training after 75 training steps without degrading stability. However this method doesn't allow trainability.