Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lethe:Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning

Hanwei Tan, Wentai Hu, Ligang He, Yijun Quan

TL;DR

This work tackles federated unlearning by exposing knowledge resurfacing as a key problem when training resumes on remaining data. It introduces Lethe, a adapter-augmented, reshape--rectify--restore pipeline that de-correlates unlearned knowledge from retained knowledge, using dual-stream updates and layer-wise rectification, followed by adapter removal and a brief recovery phase. The authors define Resurfacing Rate ($RR$) and provide first-order theoretical justification linking update correlations to forgetting persistence, supported by extensive experiments across MNIST, CIFAR-10, and Tiny-ImageNet that show near-zero resurfacing in most cases. Lethe consistently outperforms baselines in terms of forgetting persistence and efficiency, offering a practical solution for persistent FU in real-world federated deployments with sample-, class-, and client-level targets.

Abstract

Federated unlearning (FU) aims to erase designated client-level, class-level, or sample-level knowledge from a global model. Existing studies commonly assume that the collaboration ends up with the unlearning operation, overlooking the follow-up situation where the federated training continues over the remaining data.We identify a critical failure mode, termed Knowledge resurfacing, by revealing that continued training can re-activate unlearned knowledge and cause the removed influence to resurface in the global model. To address this, we propose Lethe, a novel federated unlearning method that de-correlates knowledge to be unlearned from knowledge to be retained, ensuring persistent erasure during continued training.Lethe follows a Reshape--Rectify--Restore pipeline: a temporary adapter is first trained with gradient ascent on the unlearning data to obtain magnified updates, which is then used as corrective signals to diverge layer-wise rectification on the remaining updates in two streams. Finally, the adapter is removed and a short recovery stage is performed on the retained data. Our experiments show that Lethe supports unlearning in the federated system at all levels in a unified manner and maintains superior persistence (Resurfacing Rate <1% in most cases) even after numerous rounds of follow-up training.

Lethe:Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning

TL;DR

This work tackles federated unlearning by exposing knowledge resurfacing as a key problem when training resumes on remaining data. It introduces Lethe, a adapter-augmented, reshape--rectify--restore pipeline that de-correlates unlearned knowledge from retained knowledge, using dual-stream updates and layer-wise rectification, followed by adapter removal and a brief recovery phase. The authors define Resurfacing Rate () and provide first-order theoretical justification linking update correlations to forgetting persistence, supported by extensive experiments across MNIST, CIFAR-10, and Tiny-ImageNet that show near-zero resurfacing in most cases. Lethe consistently outperforms baselines in terms of forgetting persistence and efficiency, offering a practical solution for persistent FU in real-world federated deployments with sample-, class-, and client-level targets.

Abstract

Federated unlearning (FU) aims to erase designated client-level, class-level, or sample-level knowledge from a global model. Existing studies commonly assume that the collaboration ends up with the unlearning operation, overlooking the follow-up situation where the federated training continues over the remaining data.We identify a critical failure mode, termed Knowledge resurfacing, by revealing that continued training can re-activate unlearned knowledge and cause the removed influence to resurface in the global model. To address this, we propose Lethe, a novel federated unlearning method that de-correlates knowledge to be unlearned from knowledge to be retained, ensuring persistent erasure during continued training.Lethe follows a Reshape--Rectify--Restore pipeline: a temporary adapter is first trained with gradient ascent on the unlearning data to obtain magnified updates, which is then used as corrective signals to diverge layer-wise rectification on the remaining updates in two streams. Finally, the adapter is removed and a short recovery stage is performed on the retained data. Our experiments show that Lethe supports unlearning in the federated system at all levels in a unified manner and maintains superior persistence (Resurfacing Rate <1% in most cases) even after numerous rounds of follow-up training.
Paper Structure (43 sections, 2 theorems, 36 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 43 sections, 2 theorems, 36 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Historical trajectory-based FU.
  4. FU based on modular update.
  5. FU based on update correction.
  6. Preliminaries and Problem Formulation
  7. Federated Learning and Clean Retraining
  8. Granularity of FU
  9. Problem Formulation
  10. Measure of Knowledge Resurfacing
  11. Methodology
  12. Why knowledge resurfacing happens.
  13. Theoretical analysis.
  14. The Lethe Method
  15. Theoretical analysis
  16. ...and 28 more sections

Key Result

Proposition 4.1

Let $\mathcal{L}_u(w)$ and $\mathcal{L}_r(w)$ be the losses on the unlearning set $\mathcal{D}_u$ and the remaining set $\mathcal{D}_r$. After unlearning, consider continued training on $\mathcal{D}_r$: Fix $w_{pre}$ and denote $g_u := \nabla \mathcal{L}_u(w_{pre})$. Assume: (i) $\mathcal{L}_u$ is $\beta$-smooth; (ii) $\|g_r^{(t)}\|\le G$; and (iii) there exists $\rho>0$ such that the average ali

Figures (5)

  • Figure 1: The effect of unlearning, characterized by unlearning set accuracy (lower is better), may not persist. SOTA methods proved effective in unlearning (Phase U) are susceptible to knowledge resurfacing during continued training (Phase C). The proposed Lethe (red) unlearns efficiently and stays that way (inset).
  • Figure 2: Lethe for persistent federated unlearning.Lethe takes action in the unlearning phase and follows a Reshape--Rectify--Restore pipeline: an adapter-based probe extracts $\Delta^u$ on $\mathcal{D}_u$, followed by dual-stream update rectification and adapter removal. In our scenario, federated training will subsequently resume on remaining clients, during which knowledge resurfacing is undesirable.
  • Figure 3: Visualization of the layer-wise correlation score $c^{(\ell)}=\frac{1}{T}\sum_{t=0}^{T-1}\cos\!(\Delta_r^{(t,\ell)},\,\Delta_u^{(\ell)})$ during continued federated training. Darker color indicates higher alignment of updates in the vector space. Existing algorithms fail to de-correlate the updates on the unlearning set and remaining set especially in the deep layers.
  • Figure 4: Trace of the update's correlation with the rollback vector and u-set loss during continued training (Phase C). Per-round cosine similarity $\cos(t)$ measures how the Phase C update $\Delta w_t$ aligns with the rollback vector (undoing unlearning).
  • Figure 5: Hyperparameter study of $\gamma$. Communication rounds (lower is better) versus the correlation penalty $\gamma$. Dashed lines mark the chosen values: $\gamma=0.3$ for client-level and $\gamma=1.5$ for sample-level unlearning.

Theorems & Definitions (3)

  • Proposition 4.1: Knowledge Resurfacing with Smoothness-Controlled Drift
  • Proposition 4.2: Client-update subtraction/negation suppresses correction to first order
  • proof