Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models

TL;DR

This work tackles the problem of forgetting in deep models without supervision by introducing Label-Agnostic Forgetting (LAF). LAF learns to remove information from forgotten data at the representation level using a variational framework with two VAEs to model remaining and forgetting data, coupled with an extractor unlearning loss ($L_{UE}$) and a representation alignment loss ($L_{RA}$), and it can incorporate a supervised repair step when labels are available. The approach achieves competitive results with fully supervised unlearning methods across data-removal, class-removal, and noisy-label removal tasks, and even outperforms fully supervised baselines in semi-supervised settings. This demonstrates the viability of representation-level, supervision-free unlearning and suggests a promising direction for privacy-preserving unlearning in deep models.

Abstract

Machine unlearning aims to remove information derived from forgotten data while preserving that of the remaining dataset in a well-trained model. With the increasing emphasis on data privacy, several approaches to machine unlearning have emerged. However, these methods typically rely on complete supervision throughout the unlearning process. Unfortunately, obtaining such supervision, whether for the forgetting or remaining data, can be impractical due to the substantial cost associated with annotating real-world datasets. This challenge prompts us to propose a supervision-free unlearning approach that operates without the need for labels during the unlearning process. Specifically, we introduce a variational approach to approximate the distribution of representations for the remaining data. Leveraging this approximation, we adapt the original model to eliminate information from the forgotten data at the representation level. To further address the issue of lacking supervision information, which hinders alignment with ground truth, we introduce a contrastive loss to facilitate the matching of representations between the remaining data and those of the original model, thus preserving predictive performance. Experimental results across various unlearning tasks demonstrate the effectiveness of our proposed method, Label-Agnostic Forgetting (LAF) without using any labels, which achieves comparable performance to state-of-the-art methods that rely on full supervision information. Furthermore, our approach excels in semi-supervised scenarios, leveraging limited supervision information to outperform fully supervised baselines. This work not only showcases the viability of supervision-free unlearning in deep models but also opens up a new possibility for future research in unlearning at the representation level.

Figures (6)

• Figure 1: Representation distributions in the class removal task on the FASHION dataset. The blue number 0 stand for the forgetting data while the other numbers denotes the remaining data. The colour corresponding to each class is shown in the legends.
• Figure 2: Workflow for LAF consisting of VAE training, extractor unlearning and representation alignment stages.
• Figure 3: Time cost proportion. VAE_1 stands for the training of $h$ and VAE_2 stands for the training of $h_f$
• Figure 4: Time cost comparison in the data removal task. The red columns stand for the time costs of the proposed LAF and the orange columns stand for LAF-R. The green columns denote the retraining and the blue columns denote other methods.
• Figure 5: Storage workload comparison in the data removal task. The red columns stand for the time costs of the proposed LAF and the orange columns stand for LAF-R. The green columns denote the retraining and the blue columns denote other methods.