Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forgettable Federated Linear Learning with Certified Data Unlearning

Ruinan Jin, Minghui Chen, Qiong Zhang, Xiaoxiao Li

TL;DR

This work introduces a novel FL training and unlearning strategy in DNN, termed Forgettable Federated Linear Learning (F^2L^2), and presents FedRemoval, a certified, efficient, and secure unlearning strategy that enables the server to unlearn a target client without requiring client communication or adding additional storage.

Abstract

The advent of Federated Learning (FL) has revolutionized the way distributed systems handle collaborative model training while preserving user privacy. Recently, Federated Unlearning (FU) has emerged to address demands for the "right to be forgotten"" and unlearning of the impact of poisoned clients without requiring retraining in FL. Most FU algorithms require the cooperation of retained or target clients (clients to be unlearned), introducing additional communication overhead and potential security risks. In addition, some FU methods need to store historical models to execute the unlearning process. These challenges hinder the efficiency and memory constraints of the current FU methods. Moreover, due to the complexity of nonlinear models and their training strategies, most existing FU methods for deep neural networks (DNN) lack theoretical certification. In this work, we introduce a novel FL training and unlearning strategy in DNN, termed Forgettable Federated Linear Learning (F^2L^2). F^2L^2 considers a common practice of using pre-trained models to approximate DNN linearly, allowing them to achieve similar performance as the original networks via Federated Linear Training (FLT). We then present FedRemoval, a certified, efficient, and secure unlearning strategy that enables the server to unlearn a target client without requiring client communication or adding additional storage. We have conducted extensive empirical validation on small- to large-scale datasets, using both convolutional neural networks and modern foundation models. These experiments demonstrate the effectiveness of F^2L^2 in balancing model accuracy with the successful unlearning of target clients. F^2L^2 represents a promising pipeline for efficient and trustworthy FU. The code is available here.

Forgettable Federated Linear Learning with Certified Data Unlearning

TL;DR

This work introduces a novel FL training and unlearning strategy in DNN, termed Forgettable Federated Linear Learning (F^2L^2), and presents FedRemoval, a certified, efficient, and secure unlearning strategy that enables the server to unlearn a target client without requiring client communication or adding additional storage.

Abstract

The advent of Federated Learning (FL) has revolutionized the way distributed systems handle collaborative model training while preserving user privacy. Recently, Federated Unlearning (FU) has emerged to address demands for the "right to be forgotten"" and unlearning of the impact of poisoned clients without requiring retraining in FL. Most FU algorithms require the cooperation of retained or target clients (clients to be unlearned), introducing additional communication overhead and potential security risks. In addition, some FU methods need to store historical models to execute the unlearning process. These challenges hinder the efficiency and memory constraints of the current FU methods. Moreover, due to the complexity of nonlinear models and their training strategies, most existing FU methods for deep neural networks (DNN) lack theoretical certification. In this work, we introduce a novel FL training and unlearning strategy in DNN, termed Forgettable Federated Linear Learning (F^2L^2). F^2L^2 considers a common practice of using pre-trained models to approximate DNN linearly, allowing them to achieve similar performance as the original networks via Federated Linear Training (FLT). We then present FedRemoval, a certified, efficient, and secure unlearning strategy that enables the server to unlearn a target client without requiring client communication or adding additional storage. We have conducted extensive empirical validation on small- to large-scale datasets, using both convolutional neural networks and modern foundation models. These experiments demonstrate the effectiveness of F^2L^2 in balancing model accuracy with the successful unlearning of target clients. F^2L^2 represents a promising pipeline for efficient and trustworthy FU. The code is available here.
Paper Structure (29 sections, 7 theorems, 64 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 29 sections, 7 theorems, 64 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Standard FL Algorithm
  5. Centralized Unlearning under Quadratic Loss
  6. $\mathtt{F^2L^2}$ Pipeline
  7. Federated Linear Training ($\mathtt{FLT}$)
  8. Proposed Removal ($\mathtt{FedRemoval}$)
  9. Theoretical Properties
  10. Analysis
  11. Experiments
  12. FU metrics
  13. Datasets
  14. Different FL Scenarios
  15. Experiment Details
  16. ...and 14 more sections

Key Result

Lemma 1

Suppose the local loss ${\mathcal{L}}_{c}({\bm{w}}) = {\mathcal{L}}_{\text{MSE}}({\bm{w}};{\mathcal{D}}_{c})$ is $\beta$-smooth and $\mu$-strongly convex. Let ${\bm{w}}_c^*=\mathop{\mathrm{arg\,min}}\limits {\mathcal{L}}_{c}({\bm{w}})$, ${\bm{w}}^* = \mathop{\mathrm{arg\,min}}\limits {\mathcal{L}}({ where $\sigma_{{\mathcal{L}}} = \sum_{c} p_{c}\{{\mathcal{L}}_{c}({\bm{w}}^*) -{\mathcal{L}}_{c}({\

Figures (6)

  • Figure 1: Components of FU algorithm properties considered in this paper.
  • Figure 2: Illustration of the problem setting and our proposed Forgettable Federated Linear Learning ($\mathtt{F^2L^2}$) framework. Our proposed algorithm enables the FL system to seamlessly remove a client's information from a linear model by utilizing a simple Newton's step. To achieve this, $\mathtt{FLT}$ uses a linearized DNN with FedAvg during FL. When unlearning is required for a specific client, we perform a straightforward Newton step on the model weights of the linearized DNN. To mitigate the additional communication costs associated with FL, we employ an efficient Hessian approximation in the removal step. This approach ensures both efficient unlearning and communication in FL.
  • Figure 3: Comparison to baseline FU strategies listed in Tab. \ref{['tab:baseline']} for (a) MNIST; (b) Fashion-MNIST and (c) CIFAR-10. The red and blue dashed lines indicate the TA and BSR of the retrained model, respectively. The red and blue bars represent the TA and BSR of each method, while the purple hatched bars highlight the gap between each method and retraining from scratch.
  • Figure 4: Baseline FM comparisions for (a) ImageNet (sampled); (b) DomainNet and (c) Flowers. The red and blue dashed lines indicate the TA and BSR of the retrained model, respectively. The red and blue bars represent the TA and BSR of each method, while the purple-hatched bars highlight the gap between each baseline.
  • Figure 5: The effect of regularization strength for (a) MNIST and (b) CIFAR-10 during $\mathtt{FedRemoval}$. The curve visualizes the gap between $\mathtt{FedRemoval}$ and retrained model's TA/BSR.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Lemma 1: Contraction under FedAvg algorithm (Informal)
  • Theorem 2: Removal and retrain model weight difference (informal)
  • Definition 3: $\mu$-strongly convex
  • Definition 4: $\beta$-smooth
  • Lemma 7: Relaxed triangle inequality
  • Lemma 8: Implication of $\mu$-strongly convexity and $\beta$ smoothness
  • Lemma 9: Perturbed strong convexity
  • Lemma 10
  • Lemma 11: Per-round progress of the proposed FedAvg algorithm
  • proof