Offset Unlearning for Large Language Models

TL;DR

Memorization of sensitive data in LLM training raises ethical and regulatory concerns. The authors introduce delta-Unlearning, an offset unlearning framework that uses a pair of small white-box offset models to learn a logit offset and apply it to a black-box LLM without updating its weights. On the TOFU benchmark, delta-Unlearning achieves forget performance comparable to retraining and preserves performance on out-of-forget-scope data, while remaining compatible with multiple unlearning algorithms. This approach offers privacy-preserving, modular unlearning suitable for real-world deployment, enabling application-specific knowledge control without modifying the base model.

Abstract

Despite the strong capabilities of Large Language Models (LLMs) to acquire knowledge from their training corpora, the memorization of sensitive information in the corpora such as copyrighted, biased, and private content has led to ethical and legal concerns. In response to these challenges, unlearning has emerged as a potential remedy for LLMs affected by problematic training data. However, previous unlearning techniques are either not applicable to black-box LLMs due to required access to model internal weights, or violate data protection principles by retaining sensitive data for inference-time correction. We propose δ-Unlearning, an offset unlearning framework for black-box LLMs. Instead of tuning the black-box LLM itself, δ-Unlearning learns the logit offset needed for unlearning by contrasting the logits from a pair of smaller models. Experiments demonstrate that δ- Unlearning can effectively unlearn target data while maintaining similar or even stronger performance on general out-of-forget-scope tasks. δ-Unlearning also effectively incorporates different unlearning algorithms, making our approach a versatile solution to adapting various existing unlearning algorithms to black-box LLMs.

Figures (3)

• Figure 1: Overview of $\delta$-Unlearning. In order to adapt the behavior of a black-box LLM without updating its parameters, we combine it with a pair of smaller, white-box models (which we call offset models). For unlearning, we compute the logit offset of these two models and add it to the logits of the black-box LLM given the same query. Both of the two offset models are initialized from the same checkpoint, making the logit offset zero initially. The goal of $\delta$-Unlearning is to fine-tune one of them such that their logit offset, after being added to the logits of the black-box LLM, can steer its prediction away from generating sensitive information.
• Figure 2: Unlearning trajectory of Gradient Ascent using direct fine-tuning (left), $\delta$-Unlearning (middle), and the tradeoff curve between forget and retain performance (right) over the course of unlearning. For training trajectories we report ROUGE score on all four TOFU datasets. For the tradeoff curve we report Forget Set ROUGE versus Non-forget Set ROUGE score.
• Figure 3: Effect of varying offset strength on the model after $\delta$-Unlearning with Gradient Ascent.