Not All Tokens Are Meant to Be Forgotten

TL;DR

This work addresses privacy and copyright concerns in large language models by introducing Targeted Information Forgetting (TIF), a framework that discriminates between unwanted and general information in forget samples. It combines an unwanted-information identifier with Targeted Preference Optimization (TPO), which uses Preservation Loss to keep GW intact and Logit Preference Loss to selectively reduce UW logits, mitigating over-forgetting. Empirical results on TOFU and MUSE show that TIF improves forgetting effectiveness while preserving model utility, achieving state-of-the-art performance on these benchmarks. The approach advances targeted unlearning in LLMs and lays groundwork for extending unlearning to knowledge-level scenarios while highlighting ethical considerations for responsible deployment.

Abstract

Large Language Models (LLMs), pre-trained on massive text corpora, exhibit remarkable human-level language understanding, reasoning, and decision-making abilities. However, they tend to memorize unwanted information, such as private or copyrighted content, raising significant privacy and legal concerns. Unlearning has emerged as a promising solution, but existing methods face a significant challenge of over-forgetting. This issue arises because they indiscriminately suppress the generation of all the tokens in forget samples, leading to a substantial loss of model utility. To overcome this challenge, we introduce the Targeted Information Forgetting (TIF) framework, which consists of (1) a flexible targeted information identifier designed to differentiate between unwanted words (UW) and general words (GW) in the forget samples, and (2) a novel Targeted Preference Optimization approach that leverages Logit Preference Loss to unlearn unwanted information associated with UW and Preservation Loss to retain general information in GW, effectively improving the unlearning process while mitigating utility degradation. Extensive experiments on the TOFU and MUSE benchmarks demonstrate that the proposed TIF framework enhances unlearning effectiveness while preserving model utility and achieving state-of-the-art results.

Figures (10)

• Figure 1: Illustration of the proposed TIF framework. TIF exploits an unwanted information identifier to differentiate between unwanted and general information in the forget sample (e.g., $\text{A}_\text{f}$ in the right panel). The former is represented by Unwanted Words (UW) and the latter by General Words (GW). Instead of removing the entire response $\text{A}_\text{f}$, TIF selectively unlearns only UW while preserving general knowledge associated with GW by retraining on GW. This targeted approach enables effective forgetting while maintaining model utility. The right panel demonstrates a more complete and correct model response compared to the existing unlearning approach on the left.
• Figure 2: Comparison of our TPO and NPO zhang2024negative on key metrics: forget quality and model utility. The results are derived from the Forget05 task of the TOFU dataset maini2024tofu.
• Figure 3: Illustration of responses generated by models unlearned using TPO (ours) and NPO on the retain set. The models were unlearned on the Forget05 task of the TOFU.
• Figure 4: Illustration of the proposed information identification. The discriminative approach (left) uses DistilBERT with masked token predictions, while the generative approach (right) leverages ChatGPT with explicit instructions to identify UW and GW.
• Figure 5: Model utility across various methods on the TOFU forget05 task. Each line represents evaluations conducted at every epoch (6 steps). "– GPT" denotes the use of ChatGPT-4 for unwanted information identification, while "PL" refers to the approach plus the PL Loss.