Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks

TL;DR

This work tackles class-centric machine unlearning under strict privacy constraints by introducing DELETE, a decoupled distillation method. Starting from a theoretical framework that decomposes unlearning loss into forgetting and retention components, the authors show that prior relabel-based methods primarily optimize forgetting and neglect retention, which can destabilize remaining knowledge. They propose a post-hoc masking distillation approach that uses dark knowledge from the original model to supervise the retained classes while explicitly erasing the forgotten class, formalized through masks and soft targets such as $\mathrm{KL}( \mathrm{Softmax}(\mathrm{Mask}_u'(\mathbf z)) \| \mathbf q )$. Empirically, DELETE achieves state-of-the-art forgetting/retention balance across image classification benchmarks and model families, and extends effectively to downstream tasks including face recognition, backdoor defense, and semantic segmentation, demonstrating practical applicability under data-access restrictions.

Abstract

In this work, we present DEcoupLEd Distillation To Erase (DELETE), a general and strong unlearning method for any class-centric tasks. To derive this, we first propose a theoretical framework to analyze the general form of unlearning loss and decompose it into forgetting and retention terms. Through the theoretical framework, we point out that a class of previous methods could be mainly formulated as a loss that implicitly optimizes the forgetting term while lacking supervision for the retention term, disturbing the distribution of pre-trained model and struggling to adequately preserve knowledge of the remaining classes. To address it, we refine the retention term using "dark knowledge" and propose a mask distillation unlearning method. By applying a mask to separate forgetting logits from retention logits, our approach optimizes both the forgetting and refined retention components simultaneously, retaining knowledge of the remaining classes while ensuring thorough forgetting of the target class. Without access to the remaining data or intervention (i.e., used in some works), we achieve state-of-the-art performance across various benchmarks. What's more, DELETE is a general solution that can be applied to various downstream tasks, including face recognition, backdoor defense, and semantic segmentation with great performance.

Figures (8)

• Figure 1: Performance comparison of our method with previous works and retraining (i.e., the upper bound set to 100%) across image classification, data poisoning, face recognition and semantic segmentation, evaluated on forgetting and remaining performance.
• Figure 2: Boxplot comparison of accuracy on $\mathcal{D}_{\textrm{ft}}$ (left) and $\mathcal{D}_{\textrm{rt}}$ (right) across 5 runs, illustrating accuracy distributions and stability. Ours consistently achieves the lowest $\textrm{Acc}_{\textrm{ft}}$ and the highest $\textrm{Acc}_{\textrm{rt}}$, demonstrating stable performance across executions.
• Figure 3: Accuracy comparison on $\mathcal{D}_{\textrm{ft}}$ (left) and $\mathcal{D}_{\textrm{rt}}$ (right) across multiple forgetting classes with a fixed learning rate. Ours achieves superior $\textrm{Acc}_{\textrm{ft}}$ and $\textrm{Acc}_{\textrm{rt}}$, whereas all other methods exhibit varying degrees of performance degradation.
• Figure 4: T-SNE visualization of feature representations space.
• Figure 5: Grad-CAM visualization of face recognition models on forgetting and remaining individuals.