Efficient Knowledge Deletion from Trained Models through Layer-wise Partial Machine Unlearning

TL;DR

Experimental results highlight that the partial amnesiac unlearning not only preserves model efficacy but also eliminates the necessity for brief post fine-tuning, unlike conventional amnesiac unlearning.

Abstract

Machine unlearning has garnered significant attention due to its ability to selectively erase knowledge obtained from specific training data samples in an already trained machine learning model. This capability enables data holders to adhere strictly to data protection regulations. However, existing unlearning techniques face practical constraints, often causing performance degradation, demanding brief fine-tuning post unlearning, and requiring significant storage. In response, this paper introduces a novel class of machine unlearning algorithms. First method is partial amnesiac unlearning, integration of layer-wise pruning with amnesiac unlearning. In this method, updates made to the model during training are pruned and stored, subsequently used to forget specific data from trained model. The second method assimilates layer-wise partial-updates into label-flipping and optimization-based unlearning to mitigate the adverse effects of data deletion on model efficacy. Through a detailed experimental evaluation, we showcase the effectiveness of proposed unlearning methods. Experimental results highlight that the partial amnesiac unlearning not only preserves model efficacy but also eliminates the necessity for brief post fine-tuning, unlike conventional amnesiac unlearning. Moreover, employing layer-wise partial updates in label-flipping and optimization-based unlearning techniques demonstrates superiority in preserving model efficacy compared to their naive counterparts.

Figures (4)

• Figure 1: Model behavior during training and unlearning phases. Comparison of model accuracy between partial amnesiac unlearning and conventional amnesiac unlearning on targeted and retained MNIST data for three DNN architectures: MLP (left), LeNet (middle), and ResNet$9$ (right). The vertical dashed line indicates the point of the unlearning request initiation.
• Figure 2: Model behavior during training and unlearning phases. Comparison of model accuracy between partial amnesiac unlearning and conventional amnesiac unlearning on targeted and retained OrganAMNIST data for three DNN architectures: ConvNet (left), AlexNet (middle), and ResNet$9$ (right). Vertical dashed line represents the unlearning request initiation.
• Figure 3: Class activation maps of LeNet model. Targeted classes are class-$3$ on the left side and class-$1$ on the right side. In each case, the first, second, and third column images are the class activation maps before unlearning, after conventional amnesiac unlearning, and proposed partial amnesiac unlearning, respectively. Blue colour indicates low activation region while red colur indicates high activation regions.
• Figure 4: Comparison of model efficacy between partial amnesiac unlearning between conventional amnesiac unlearning against percentage of affected batches.