Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computation and Communication Efficient Federated Unlearning via On-server Gradient Conflict Mitigation and Expression

Minh-Duong Nguyen, Senura Hansaja, Le-Tuan Nguyen, Quoc-Viet Pham, Ken-Tye Yong, Nguyen H. Tran, Dung D. Le

Abstract

Federated Unlearning (FUL) aims to remove specific participants' data contributions from a trained Federated Learning model, thereby ensuring data privacy and compliance with regulatory requirements. Despite its potential, progress in FUL has been limited due to several challenges, including the cross-client knowledge inaccessibility and high computational and communication costs. To overcome these challenges, we propose Federated On-server Unlearning (FOUL), a novel framework that comprises two key stages. The learning-to-unlearn stage serves as a preparatory learning phase, during which the model identifies and encodes the key features associated with the forget clients. This stage is communication-efficient and establishes the basis for the subsequent unlearning process. Subsequently, on-server knowledge aggregation phase aims to perform the unlearning process at the server without requiring access to client data, thereby preserving both efficiency and privacy. We introduce a new data setting for FUL, which enables a more transparent and rigorous evaluation of unlearning. To highlight the effectiveness of our approach, we propose a novel evaluation metric termed time-to-forget, which measures how quickly the model achieves optimal unlearning performance. Extensive experiments conducted on three datasets under various unlearning scenarios demonstrate that FOUL outperforms the Retraining in FUL. Moreover, FOUL achieves competitive or superior results with significantly reduced time-to-forget, while maintaining low communication and computation costs.

Computation and Communication Efficient Federated Unlearning via On-server Gradient Conflict Mitigation and Expression

Abstract

Federated Unlearning (FUL) aims to remove specific participants' data contributions from a trained Federated Learning model, thereby ensuring data privacy and compliance with regulatory requirements. Despite its potential, progress in FUL has been limited due to several challenges, including the cross-client knowledge inaccessibility and high computational and communication costs. To overcome these challenges, we propose Federated On-server Unlearning (FOUL), a novel framework that comprises two key stages. The learning-to-unlearn stage serves as a preparatory learning phase, during which the model identifies and encodes the key features associated with the forget clients. This stage is communication-efficient and establishes the basis for the subsequent unlearning process. Subsequently, on-server knowledge aggregation phase aims to perform the unlearning process at the server without requiring access to client data, thereby preserving both efficiency and privacy. We introduce a new data setting for FUL, which enables a more transparent and rigorous evaluation of unlearning. To highlight the effectiveness of our approach, we propose a novel evaluation metric termed time-to-forget, which measures how quickly the model achieves optimal unlearning performance. Extensive experiments conducted on three datasets under various unlearning scenarios demonstrate that FOUL outperforms the Retraining in FUL. Moreover, FOUL achieves competitive or superior results with significantly reduced time-to-forget, while maintaining low communication and computation costs.
Paper Structure (41 sections, 1 theorem, 25 equations, 11 figures, 4 tables, 2 algorithms)

This paper contains 41 sections, 1 theorem, 25 equations, 11 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations:
  4. Federated Unlearning
  5. Methods
  6. Learning to unlearn
  7. Motivation
  8. Learning for Representation Featurizer
  9. General learning framework.
  10. Training Causal Featurizer.
  11. Training Non-causal Featurizer.
  12. Meaningful Representations.
  13. Client-wise Unlearning
  14. On-server gradient-based unlearning
  15. Experimental Setup
  16. ...and 26 more sections

Key Result

Theorem 1

Given $\Gamma = \{\gamma^{(r)}_{u}\vert u\in{\mathcal{U}}, \sum_{u\in {\mathcal{U}}}\gamma^{(r)}_u = 1\}$ is the set of learnable coefficients at each round $r$, where ${\mathcal{U}} = \{{\mathcal{U}}_{\mathcal{R}}, {\mathcal{U}}_{\mathcal{F}}\}$. Optimal unlearning gradient $\nabla^{(r)}_\textrm{FO

Figures (11)

  • Figure 1: Extension of the causal structure model to the multi-client setting. Following lu2021invariant, the causal factors ${\mathcal{Z}}_K$ represent the domain-invariant characteristics. Consequently, when unlearning a client, removing ${\mathcal{Z}}_K$ may reduce the amount of useful information necessary for predicting ${\mathcal{Y}}$. In contrast, unlearning the non-causal factors ${\mathcal{Z}}_{V,2}$ primarily removes domain-specific information from client 2, while having a smaller impact on the information relevant to the classification of ${\mathcal{Y}}$. Dashed lines denote edges that may vary across clients or be absent in certain scenarios, solid lines indicate edges that remain invariant across all clients.
  • Figure 2: The learning on a single local client in the L2U stage. Featurizers are trained to disentangle knowledge into invariant and variant representations.
  • Figure 3: Unlearning stage with data-free on-server unlearning. Only the non-causal featurizer participates in this phase. The gradients of the non-causal featurizer from both retain and forget users are sent to the server for on-server gradient matching.
  • Figure 4: Retraining from scratch with FedAvg without resetting model parameters on PACS dataset.
  • Figure 5: Retraining from scratch with FedAvg with resetting model parameters on PACS dataset.
  • ...and 6 more figures

Theorems & Definitions (8)

  • Definition 1: Forget user set
  • Definition 2: Retain user set
  • Definition 3: Learning Stage
  • Definition 4: Unlearning Stage
  • Definition 5: Causal Structured Model 2022-DG-CIRL
  • Definition 6: Causal and non-causal representations 2022-DG-CIRL
  • Definition 7: Causal Principle in Unlearning
  • Theorem 1: FOUL solution