Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Unlearning with Knowledge Distillation

Chen Wu, Sencun Zhu, Prasenjit Mitra

TL;DR

The paper tackles the challenge of forgetting a designated client's contribution in federated learning without requiring client participation or data. It first removes the client's historical updates from the final global model and then mitigates resulting skew via knowledge distillation using unlabeled server-side data. Backdoor attacks are used to evaluate unlearning effectiveness, showing that subtraction eliminates the attacker's influence while distillation recovers predictive performance and preserves privacy. The approach offers a practical, server-side solution with strong efficiency advantages over retraining from scratch, validated across three datasets and multiple architectures.

Abstract

Federated Learning (FL) is designed to protect the data privacy of each client during the training process by transmitting only models instead of the original data. However, the trained model may memorize certain information about the training data. With the recent legislation on right to be forgotten, it is crucially essential for the FL model to possess the ability to forget what it has learned from each client. We propose a novel federated unlearning method to eliminate a client's contribution by subtracting the accumulated historical updates from the model and leveraging the knowledge distillation method to restore the model's performance without using any data from the clients. This method does not have any restrictions on the type of neural networks and does not rely on clients' participation, so it is practical and efficient in the FL system. We further introduce backdoor attacks in the training process to help evaluate the unlearning effect. Experiments on three canonical datasets demonstrate the effectiveness and efficiency of our method.

Federated Unlearning with Knowledge Distillation

TL;DR

The paper tackles the challenge of forgetting a designated client's contribution in federated learning without requiring client participation or data. It first removes the client's historical updates from the final global model and then mitigates resulting skew via knowledge distillation using unlabeled server-side data. Backdoor attacks are used to evaluate unlearning effectiveness, showing that subtraction eliminates the attacker's influence while distillation recovers predictive performance and preserves privacy. The approach offers a practical, server-side solution with strong efficiency advantages over retraining from scratch, validated across three datasets and multiple architectures.

Abstract

Federated Learning (FL) is designed to protect the data privacy of each client during the training process by transmitting only models instead of the original data. However, the trained model may memorize certain information about the training data. With the recent legislation on right to be forgotten, it is crucially essential for the FL model to possess the ability to forget what it has learned from each client. We propose a novel federated unlearning method to eliminate a client's contribution by subtracting the accumulated historical updates from the model and leveraging the knowledge distillation method to restore the model's performance without using any data from the clients. This method does not have any restrictions on the type of neural networks and does not rely on clients' participation, so it is practical and efficient in the FL system. We further introduce backdoor attacks in the training process to help evaluate the unlearning effect. Experiments on three canonical datasets demonstrate the effectiveness and efficiency of our method.
Paper Structure (16 sections, 8 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 16 sections, 8 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. Federated Learning and Unlearning
  5. Challenges in Federated Unlearning
  6. Evaluation of Federated Unlearning
  7. Unlearning Method
  8. Erase Historical Parameter Updates
  9. Remedy with Knowledge Distillation
  10. Experiments
  11. Overview of the Experimental Setup
  12. Dataset Description
  13. Model Architectures
  14. Unlearning Target with Backdoor Attack
  15. Unlearning Model Performance Evaluation
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Samples of backdoor images targeting digit 1 in MNIST, class of truck in CIFAR-10, and the class of stop sign in GTSRB dataset.
  • Figure 2: The training process and the corresponding subtraction unlearning process with MNIST dataset. The solid lines stand for the test accuracy and backdoor attack success rate in the training process. The dotted lines represent the test accuracy and backdoor attack success rate of the unlearning model after merely subtracting the target historical parameter updates.
  • Figure 3: The performance of the unlearning model under knowledge distillation training process with MNIST dataset. The blue line stands for the change of losses which is calculated using the loss of the original model before subtracting historical parameter updates from the target client divided by the loss of the unlearning model. It means the model is better recovered with a value closer to 1.