Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation

Abstract

With the increasing importance of data privacy and security, federated unlearning has emerged as a novel research field dedicated to ensuring that federated learning models no longer retain or leak relevant information once specific data has been deleted. In this paper, to the best of our knowledge, we propose the first complete pipeline for federated unlearning, which includes a federated unlearning approach and an evaluation framework. Our proposed federated unlearning approach ensures high efficiency and model accuracy without the need to store historical data.It effectively leverages the knowledge distillation model alongside various optimization mechanisms. Moreover, we propose a framework named Skyeye to visualize the forgetting capacity of federated unlearning models. It utilizes the federated unlearning model as the classifier integrated into a Generative Adversarial Network (GAN). Afterward, both the classifier and discriminator guide the generator in generating samples. Throughout this process, the generator learns from the classifier's knowledge. The generator then visualizes this knowledge through sample generation. Finally, the model's forgetting capability is evaluated based on the relevance between the deleted data and the generated samples. Comprehensive experiments are conducted to illustrate the effectiveness of the proposed federated unlearning approach and the corresponding evaluation framework.

Figures (14)

• Figure 1: Proposed federated unlearning approach. When the server receives a user's deletion request, the following steps are executed sequentially on the server side: ① Initialization of the teacher model and the student model. It takes the current global model as the student model and an incompetent model as the teacher model. ② Loss Calculation. The total loss consists of distillation loss, attention map alignment loss, and hard loss. ③ Improvements of training with different mechanisms. To further enhance the training process, we propose mechanisms for handling loss explosion, preventing catastrophic forgetting, and ensuring performance recovery.
• Figure 2: Skyeye framework of forgetting capability evaluation. The main procedures are described below. ① Users send data deletion requests to the server. ② Server executes the unlearning algorithm. ③ Users download the unlearned model. ④ Integrate the unlearned model into the GAN network. ⑤ Training Phase. The generator takes random noise samples and random labels as input to generate fake samples, while the discriminator is trained to differentiate between real and fake samples. The classifier receives the same random labels and fake samples to classify them into different categories. Ultimately, both the classifier and discriminator work together to guide the generator in generating samples. ⑥ Decision-making Phase. Users evaluate the server's unlearning algorithm by assessing the relevance between the images generated by the generator and the deleted data.
• Figure 3: (a) Accuracy Rate, (b) Success Rate of Backdoor Attack, and (c) Success Rate of Membership Inference Attack of models on the MNIST dataset.
• Figure 4: (a) Accuracy Rate, (b) Success Rate of Backdoor Attack, and (c) Success Rate of Membership Inference Attack of models on the CIFAR-10 dataset.
• Figure 5: (a) Accuracy Rate, (b) Success Rate of Backdoor Attack, and (c) Success Rate of Membership Inference Attack of models on the CIFAR-100 dataset.