Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter

TL;DR

The paper tackles federated unlearning (FU) by addressing indiscriminate forgetting, irreversibility, and high costs in FL. It introduces FUSED, a reversible FU method that first identifies knowledge-sensitive layers via Critical Layer Identification (CLI) and then trains independent sparse adapters to overwrite forgotten knowledge while freezing the base model, enabling reversibility by removing adapters. The approach achieves unlearning performance comparable to retraining while significantly reducing communication, computation, and storage costs, and it demonstrates robustness against privacy leakage in various FU scenarios (client, class, and sample). Theoretical analysis explains how partial, sparsified updates can overwrite old knowledge, and extensive experiments across multiple datasets and models validate effectiveness, efficiency, and reversibility. Overall, FUSED offers a practical, privacy-preserving, and memory-efficient FU framework for federated systems and opens avenues for knowledge editing in FL.

Abstract

Federated Learning is a promising paradigm for privacy-preserving collaborative model training. In practice, it is essential not only to continuously train the model to acquire new knowledge but also to guarantee old knowledge the right to be forgotten (i.e., federated unlearning), especially for privacy-sensitive information or harmful knowledge. However, current federated unlearning methods face several challenges, including indiscriminate unlearning of cross-client knowledge, irreversibility of unlearning, and significant unlearning costs. To this end, we propose a method named FUSED, which first identifies critical layers by analyzing each layer's sensitivity to knowledge and constructs sparse unlearning adapters for sensitive ones. Then, the adapters are trained without altering the original parameters, overwriting the unlearning knowledge with the remaining knowledge. This knowledge overwriting process enables FUSED to mitigate the effects of indiscriminate unlearning. Moreover, the introduction of independent adapters makes unlearning reversible and significantly reduces the unlearning costs. Finally, extensive experiments on three datasets across various unlearning scenarios demonstrate that FUSED's effectiveness is comparable to Retraining, surpassing all other baselines while greatly reducing unlearning costs.

Figures (3)

• Figure 1: The figure illustrates the process of CLI (left) and unlearning (right). Left: the server computes the difference of each layer between the models uploaded by each client and the distributed one, identifying critical layers that are sensitive to knowledge. Right: a sparse adapter is constructed for each key layer, which is then independently trained on the remaining data.
• Figure 2: The average difference between local models and the server model across different models.
• Figure 3: Ablation study of CLI.