Representation Noising: A Defence Mechanism Against Harmful Finetuning

TL;DR

The paper tackles the risk of harmful fine-tuning in LLMs, including scenarios where attackers obtain weights or fine-tuning APIs. It introduces Representation Noising (RepNoise), a layer-wise training objective that simultaneously suppresses harmful information, preserves harmless task performance, and pushes harmful representations toward noise, using a distributional distance (MMD) to approximate harmful information. The authors frame the defense via an information-theoretic lens, linking mutual information reductions to increased resistance against HFAs, and validate the method through extensive experiments across resistance, stability, trainability, and generalization, with mechanistic analyses showing deep, cross-layer information removal as key. RepNoise achieves strong in-distribution defense against harmful QA and toxicity fine-tuning while preserving model capabilities on benign tasks, albeit with limitations in cross-domain generalization and notable hyperparameter sensitivity. The work highlights both the potential and current constraints of immunizing LLMs against HFAs, offering a principled approach that complements existing safety measures.

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.

Figures (9)

• Figure 1: Representation Noising pushes the intermediate activations of harmful text inputs (their representations) towards random directions, effectively reducing the mutual information between harmful representations and harmful text sequences and making it difficult to recover harmful representations through HFAs. We visualize this here as a projection (PCA) which isn't able to recover any structure.
• Figure 2: $L_2$ distance between weights of each layer between the base model, a successfully attacked model and two defences. RepNoise's differences spread through the layers compared to Adversarial loss where the weight differences are concentrated at the later layers indicative of superficial defence.
• Figure 3: Log probability of harmful and harmless sequences across layers. Notice how adversarial loss mostly depromotes harmful tokens towards the last layer. This is done more evenly across layers for RepNoise indicating comprehensive and deep information removal.
• Figure 4: PCA across 100 harmful and harmless samples from BeaverTails on the activations of the last layer.
• Figure 5: Harmful probe accuracy on (a) base model and attacked model, (b) base model and models trained with RepNoise ($\beta=4$) and adversarial Loss, and (c) base model, RepNoise model and an attacked RepNoise model

Theorems & Definitions (2)

• Theorem 1
• Theorem 2