Efficient Verified Machine Unlearning For Distillation

TL;DR

The paper tackles the challenge of efficiently and verifiably removing data influence in knowledge-distillation pipelines, where teacher-to-student information flow undermines traditional SISA-based unlearning. It introduces PURGE, a framework that partitions teacher constituents among disjoint student shards and uses incremental multi-teacher distillation to preserve data isolation during training. The authors provide theoretical speed-up analyses showing at least $N$-fold improvements over naive baselines for teacher-side unlearning, while empirical results demonstrate that PURGE maintains student accuracy close to SISA across vision, language, and mixed tasks. This approach enables practical, privacy-preserving deployment of distillation-based systems by significantly reducing retraining costs without sacrificing performance.

Abstract

Growing data privacy demands, driven by regulations like GDPR and CCPA, require machine unlearning methods capable of swiftly removing the influence of specific training points. Although verified approaches like SISA, using data slicing and checkpointing, achieve efficient unlearning for single models by reverting to intermediate states, these methods struggle in teacher-student knowledge distillation settings. Unlearning in the teacher typically forces costly, complete student retraining due to pervasive information propagation during distillation. Our primary contribution is PURGE (Partitioned Unlearning with Retraining Guarantee for Ensembles), a novel framework integrating verified unlearning with distillation. We introduce constituent mapping and an incremental multi-teacher strategy that partitions the distillation process, confines each teacher constituent's impact to distinct student data subsets, and crucially maintains data isolation. The PURGE framework substantially reduces retraining overhead, requiring only partial student updates when teacher-side unlearning occurs. We provide both theoretical analysis, quantifying significant speed-ups in the unlearning process, and empirical validation on multiple datasets, demonstrating that PURGE achieves these efficiency gains while maintaining student accuracy comparable to standard baselines.

Figures (6)

• Figure 1: Overview of the proposed framework (PURGE) integrating SISA with knowledge distillation for efficient, verified unlearning. The structure maintains data isolation during distillation, enabling efficient student retraining upon teacher updates. Key steps: (1) Sharding: Student data $\mathcal{D}^S$ is partitioned into $N$ shards ($\mathcal{D}^S_k$), each assigned to a student constituent model ($\mathcal{S}_k$). (2) Mapping: Each $\mathcal{S}_k$ is mapped to a distinct teacher ensemble $\mathscr{T}_k = \{\mathcal{T}_{k,1}, ..., \mathcal{T}_{k,c_k}\}$. (3) Incremental Distillation: The student shard $\mathcal{D}^S_k$ is processed in $c_k$ sequential chunks ($\mathcal{D}^S_{k,l}$). Crucially, soft labels ($Y_{k,l}$) for chunk $l$ are generated only by an incrementally growing teacher subensemble $\mathscr{T}_{k,l} = \cup_{i \in [l]} \mathcal{T}_{k,i}$, limiting information propagation from the full teacher ensemble. (4) SISA Slicing & Training: Each resulting data-label chunk $\mathcal{D}^\dagger_{k,l}=[\mathcal{D}_{k,l}^S, Y_{k,l}]$ is further divided into $R_l$ slices ($\mathcal{D}^\dagger_{k,l,j}$). $\mathcal{S}_k$ trains incrementally on these slices using standard SISA checkpointing. (5) Aggregation: Final predictions are aggregated from all trained student constituent models $\{\mathcal{S}_k\}$. This design ensures that unlearning affecting teacher $\mathcal{T}_{k,i}$ only requires partial retraining of the corresponding student $\mathcal{S}_k$.
• Figure 2: Speed comparison of the student network update process for 100 unlearning requests sent to the teacher network with 32 constituent models ($M=32$) on MNIST dataset. Top row: $r=1$; bottom row: $r=4$ slices per chunk. Left column: cumulative processing time. Right column: measured average speed-up over naive SISA (red curve follows Eq. \ref{['eqn:speed_up_M']}).
• Figure 3: Comparison of student network accuracy on MNIST (top row) and SVHN (bottom row). Accuracy is plotted against the number of student constituent models ($N$) for different teacher ensemble sizes ($M=8, 16, 32$). The plot shows results for PURGE, the SISA baseline student, the original Teacher ensemble, and the Single-teacher Soft Label ablation.
• Figure 4: Loss curve comparison between multi-teacher soft label generation and single-teacher ablation
• Figure 5: Comparison of student network accuracy on $10\%$, $20\%$ and $50\%$ versions of MNIST and SVHN. The plot shows results for PURGE, the SISA baseline student, the original Teacher ensemble, and the Single-teacher Soft Label ablation. The original Teacher ensemble was trained on the full training sets, while the student networks are trained on the corresponding subsets.