QuickDrop: Efficient Federated Unlearning by Integrated Dataset Distillation

TL;DR

This work tackles the challenge of data forgetting in federated learning where data remains on client devices. It introduces QuickDrop, a method that synthesizes compact, per-client datasets via gradient matching during FL training and uses these synthetic sets to perform unlearning with fast stochastic gradient ascent, while recovery and relearning reuse the same synthetic data. The approach formulates a distillation objective specific to unlearning, enabling efficient gradient-based forgetting with negligible data movement and strong privacy properties. Empirical results on MNIST, CIFAR-10, and SVHN across up to 100 clients show substantial speedups, up to 463x faster than retraining-from-scratch and 65x faster than baselines, with competitive accuracy and robust performance under sequential unlearning requests. QuickDrop thus provides a practical, scalable solution for privacy-preserving federated unlearning with real-world applicability.

Abstract

Federated Unlearning (FU) aims to delete specific training data from an ML model trained using Federated Learning (FL). We introduce QuickDrop, an efficient and original FU method that utilizes dataset distillation (DD) to accelerate unlearning and drastically reduces computational overhead compared to existing approaches. In QuickDrop, each client uses DD to generate a compact dataset representative of the original training dataset, called a distilled dataset, and uses this compact dataset during unlearning. To unlearn specific knowledge from the global model, QuickDrop has clients execute Stochastic Gradient Ascent with samples from the distilled datasets, thus significantly reducing computational overhead compared to conventional FU methods. We further increase the efficiency of QuickDrop by ingeniously integrating DD into the FL training process. By reusing the gradient updates produced during FL training for DD, the overhead of creating distilled datasets becomes close to negligible. Evaluations on three standard datasets show that, with comparable accuracy guarantees, QuickDrop reduces the duration of unlearning by 463.8x compared to model retraining from scratch and 65.1x compared to existing FU approaches. We also demonstrate the scalability of QuickDrop with 100 clients and show its effectiveness while handling multiple unlearning operations.

Figures (6)

• Figure 1: The workflow of QuickDrop, our efficient federated unlearning method using synthetic data.
• Figure 2: Class-wise testing accuracy on the CIFAR-10 dataset when unlearning class $9$. QuickDrop successfully unlearns class $9$ while retaining good performance on the remaining classes after recovery.
• Figure 3: The membership inference attack (MIA) accuracy of all baselines after unlearning on the CIFAR-10 dataset with $10$ clients and non-IID partitioning.
• Figure 4: The accuracy of each class with sequential unlearning requests on CIFAR-10 and with $\alpha=0.1$. We unlearn a random class every five rounds and highlight the unlearning and recovery stages in blue and yellow shades, respectively. The class unlearning order is $[5, 8, 0, 3, 2, 4, 7, 9, 1, 6]$.
• Figure 5: The accuracy on the R-Set after recovery (left) and the number of gradient computations performed on original data (right) when doing additional fine-tuning steps on CIFAR-10. The portion in orange corresponds to FL training, while the gray portion corresponds to new gradients computed for fine-tuning. QuickDrop nearly matches the accuracy of Retrain-Or at an extra gradient cost no higher than that of FL training.