FIT: Defying Catastrophic Forgetting in Continual LLM Unlearning

TL;DR

fit is introduced, a framework for continual unlearning that handles large numbers of deletion requests while maintaining robustness against both catastrophic forgetting and post-unlearning recovery, and surpasses existing methods on MMLU, CommonsenseQA, and GSM8K and remains resistant against both relearning and quantization recovery attacks.

Abstract

Large language models (LLMs) demonstrate impressive capabilities across diverse tasks but raise concerns about privacy, copyright, and harmful materials. Existing LLM unlearning methods rarely consider the continual and high-volume nature of real-world deletion requests, which can cause utility degradation and catastrophic forgetting as requests accumulate. To address this challenge, we introduce \fit, a framework for continual unlearning that handles large numbers of deletion requests while maintaining robustness against both catastrophic forgetting and post-unlearning recovery. \fit mitigates degradation through rigorous data \underline{F}iltering, \underline{I}mportance-aware updates, and \underline{T}argeted layer attribution, enabling stable performance across long sequences of unlearning operations and achieving a favorable balance between forgetting effectiveness and utility retention. To support realistic evaluation, we present \textbf{PCH}, a benchmark covering \textbf{P}ersonal information, \textbf{C}opyright, and \textbf{H}armful content in sequential deletion scenarios, along with two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), which jointly assess forgetting quality and utility preservation. Extensive experiments on four open-source LLMs with hundreds of deletion requests show that \fit achieves the strongest trade-off between F.D. and R.U., surpasses existing methods on MMLU, CommonsenseQA, and GSM8K, and remains resistant against both relearning and quantization recovery attacks.

Figures (18)

• Figure 1: Left: Schematics of single-shot vs. continual unlearning; Right: Retain and forget accuracy on Yi-6B using GA for single unlearning and $100$ sequential request(s), and catastrophic forgetting begins after roughly $25$ requests.
• Figure 2: Overview of FIT: Incoming unlearning requests are first de-duplicated via embedding-based redundancy filtering (Section \ref{['subsec:redundancy_filtering']}). For each filtered request, an importance score then guides adaptive selection of the unlearning method (Section \ref{['subsec:algorithm_selection']}), and targeted layer attribution restricts updates to the top‑$K$ influential layers (Section \ref{['subsec:layer_attribution']}), mitigating compounded knowledge loss and parameter drift.
• Figure 3: Estimated decay of shared-token probabilities as semantically similar requests are iteratively removed: Redundant gradients push them toward near zero, causing model collapse, whereas effective unlearning stabilizes them at moderate, non-zero levels.
• Figure 4: Performance of the unlearning methods across importance levels: Rows trace the forgetting and retain accuracy curves; columns represent the low, medium, and high. Our goal is to select unlearning algorithms for low forget and high retain accuracy.
• Figure 5: Histogram of layer selection: Each bar shows the frequency with which the corresponding layer was chosen.