ESC: Erasing Space Concept for Knowledge Deletion

TL;DR

This work reframes machine unlearning as Knowledge Deletion (KD), addressing real-world demands for removing both utility-relevant and privacy-sensitive knowledge from models. It identifies a critical gap: prior MU methods preserve feature-level knowledge that can leak or be exploited, and thus proposes a feature-space deletion approach. The authors introduce Erasing Space Concept (ESC) and its training-enabled variant ESC-T to directly suppress forgetting information in embedding spaces, with Knowledge Retention Score (KR) as a principled metric for feature-level knowledge retention. Through extensive experiments across diverse datasets and models, ESC/ESC-T achieve fast, state-of-the-art KD performance and robust KR scores, including in facial-domain scenarios, suggesting practical utility for privacy-preserving model updates. The work also demonstrates efficiency, broad applicability, and provides a pathway for future extensions to larger models and generative domains.

Abstract

As concerns regarding privacy in deep learning continue to grow, individuals are increasingly apprehensive about the potential exploitation of their personal knowledge in trained models. Despite several research efforts to address this, they often fail to consider the real-world demand from users for complete knowledge erasure. Furthermore, our investigation reveals that existing methods have a risk of leaking personal knowledge through embedding features. To address these issues, we introduce a novel concept of Knowledge Deletion (KD), an advanced task that considers both concerns, and provides an appropriate metric, named Knowledge Retention score (KR), for assessing knowledge retention in feature space. To achieve this, we propose a novel training-free erasing approach named Erasing Space Concept (ESC), which restricts the important subspace for the forgetting knowledge by eliminating the relevant activations in the feature. In addition, we suggest ESC with Training (ESC-T), which uses a learnable mask to better balance the trade-off between forgetting and preserving knowledge in KD. Our extensive experiments on various datasets and models demonstrate that our proposed methods achieve the fastest and state-of-the-art performance. Notably, our methods are applicable to diverse forgetting scenarios, such as facial domain setting, demonstrating the generalizability of our methods. The code is available at http://github.com/KU-VGI/ESC .

Figures (14)

• Figure 1: The recovery rate of each unlearning method using All-CNN in CIFAR-10. We conducted linear probing with each unlearned feature extractor and obtain the accuracy on the forgetting training data. Knowledge retention rate is calculated as $acc_{f}^R/acc_{f}^O$, where $acc_{f}^R$ is the accuracy after linear probing, and $acc_{f}^O$ is the accuracy of the original model. While existing unlearning methods still retain the forgetting knowledge, our methods effectively eliminate it from the feature representation.
• Figure 2: Using CIFAR-10 krizhevsky2009learning with All-CNN springenberg2014striving, we measure weight difference between the original model and unlearned model. We calculate the difference as $\gamma\cdot\Vert\theta^l_{ori}-\theta^l_{ul}\Vert/\Vert\theta^l_{ori}$, where $\theta$ means weight of model and $\gamma$ is a scaling factor that reflects how the change in a specific layer contributes to the overall change in the model. The change in the head is dominant over the change in the entire model for all MU methods. This implies that most models focus on the head when performing MU.
• Figure 3: We calculate the cosine similarity between features for each class. (Left) Cosine similarity between the original features. (Right) Cosine similarity between the original features and the ESC features, where the principal component has been removed from the original features.
• Figure 4: An overview of our methods. (a) We start with extracting the principal directions ($\mathbf{U}$) of embedding features from the forgetting data using SVD. (b) ESC erases crucial directions from $\mathbf{U}$, and we can get the pruned principal directions $\mathbf{U}_P$. (c) ESC-T enhances ESC by incorporating our forgetting loss, $\mathcal{L}_{PCE}$, refining the erasure process to eliminate only important elements rather than entire directions. ESC-T yields the refined principal directions $\mathbf{U}_R$, which improve the trade-off forgetting and preservation. (d) During the inference phase, we project the extracted features onto the subspace formed by $\mathbf{U}_P$ or $\mathbf{U}_R$ from each method.
• Figure 5: Comparison of time consumption. Our approaches are the most efficient method compared with others. Additional results reported in Supplementary \ref{['supp:add_exp']}.