Subspace based Federated Unlearning

TL;DR

The paper tackles federated unlearning under storage constraints by introducing SFU, a subspace-based approach that restricts the target client's gradient ascent to the orthogonal subspace of input gradients from other clients, avoiding the need to store historical updates. It computes per-layer representation matrices via SVD from other clients and protects them with differential privacy, enabling a projection-based forgetting update on the server. SFU demonstrates strong forgetting performance with minimal accuracy loss and fast recovery, outperforming Retraining, UL, and GA across MNIST, CIFAR10, and CIFAR100. This approach offers a practical, privacy-conscious solution for forgetting in FL with limited storage and communication overhead.

Abstract

Federated learning (FL) enables multiple clients to train a machine learning model collaboratively without exchanging their local data. Federated unlearning is an inverse FL process that aims to remove a specified target client's contribution in FL to satisfy the user's right to be forgotten. Most existing federated unlearning algorithms require the server to store the history of the parameter updates, which is not applicable in scenarios where the server storage resource is constrained. In this paper, we propose a simple-yet-effective subspace based federated unlearning method, dubbed SFU, that lets the global model perform gradient ascent in the orthogonal space of input gradient spaces formed by other clients to eliminate the target client's contribution without requiring additional storage. Specifically, the server first collects the gradients generated from the target client after performing gradient ascent, and the input representation matrix is computed locally by the remaining clients. We also design a differential privacy method to protect the privacy of the representation matrix. Then the server merges those representation matrices to get the input gradient subspace and updates the global model in the orthogonal subspace of the input gradient subspace to complete the forgetting task with minimal model performance degradation. Experiments on MNIST, CIFAR10, and CIFAR100 show that SFU outperforms several state-of-the-art (SOTA) federated unlearning algorithms by a large margin in various settings.

Figures (4)

• Figure 1: The pipeline of the SFU.The whole process takes place after the FL model has been trained. The orange client represents the target client whose contribution is to be removed; The blue ones represent other clients. The boxes on the right of the image represent global model updates that happen on the server.
• Figure 2: Model accuracy after execution of SFU and other baselines at various stages of FL model training.
• Figure 4: Convergence plots for SFU and other baselines in different datasets with CNN.
• Figure 5: Heat maps for each client with CIFAR10 dataset under different data partitions. The color bar denotes the number of data samples. Each rectangle represents the number of data samples of a specific class in a party.