Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Machine Unlearning via Null Space Calibration

Huiqiang Chen, Tianqing Zhu, Xin Yu, Wanlei Zhou

TL;DR

This work addresses the challenge of forgetting specific data in trained models without degrading performance on remaining data. It introduces UNSC, which confines unlearning to a null space defined by remaining data and uses pseudo-labeling to steer unlearning samples toward plausible remaining classes, thereby reducing over-unlearning and potentially boosting accuracy on $\\mathcal{D}_r$. Theoretical results justify why null-space updates preserve performance on remaining data, while empirical results across multiple datasets show UNSC outperforms or matches retraining with less degradation of utility and stronger privacy guarantees. This approach offers a practical, scalable pathway for compliant machine learning systems facing deletion requests and memory constraints.

Abstract

Machine unlearning aims to enable models to forget specific data instances when receiving deletion requests. Current research centres on efficient unlearning to erase the influence of data from the model and neglects the subsequent impacts on the remaining data. Consequently, existing unlearning algorithms degrade the model's performance after unlearning, known as \textit{over-unlearning}. This paper addresses this critical yet under-explored issue by introducing machine \underline{U}nlearning via \underline{N}ull \underline{S}pace \underline{C}alibration (UNSC), which can accurately unlearn target samples without over-unlearning. On the contrary, by calibrating the decision space during unlearning, UNSC can significantly improve the model's performance on the remaining samples. In particular, our approach hinges on confining the unlearning process to a specified null space tailored to the remaining samples, which is augmented by strategically pseudo-labeling the unlearning samples. Comparative analyses against several established baselines affirm the superiority of our approach. Code is released at this \href{https://github.com/HQC-ML/Machine-Unlearning-via-Null-Space-Calibration}{URL}.

Machine Unlearning via Null Space Calibration

TL;DR

This work addresses the challenge of forgetting specific data in trained models without degrading performance on remaining data. It introduces UNSC, which confines unlearning to a null space defined by remaining data and uses pseudo-labeling to steer unlearning samples toward plausible remaining classes, thereby reducing over-unlearning and potentially boosting accuracy on . Theoretical results justify why null-space updates preserve performance on remaining data, while empirical results across multiple datasets show UNSC outperforms or matches retraining with less degradation of utility and stronger privacy guarantees. This approach offers a practical, scalable pathway for compliant machine learning systems facing deletion requests and memory constraints.

Abstract

Machine unlearning aims to enable models to forget specific data instances when receiving deletion requests. Current research centres on efficient unlearning to erase the influence of data from the model and neglects the subsequent impacts on the remaining data. Consequently, existing unlearning algorithms degrade the model's performance after unlearning, known as \textit{over-unlearning}. This paper addresses this critical yet under-explored issue by introducing machine \underline{U}nlearning via \underline{N}ull \underline{S}pace \underline{C}alibration (UNSC), which can accurately unlearn target samples without over-unlearning. On the contrary, by calibrating the decision space during unlearning, UNSC can significantly improve the model's performance on the remaining samples. In particular, our approach hinges on confining the unlearning process to a specified null space tailored to the remaining samples, which is augmented by strategically pseudo-labeling the unlearning samples. Comparative analyses against several established baselines affirm the superiority of our approach. Code is released at this \href{https://github.com/HQC-ML/Machine-Unlearning-via-Null-Space-Calibration}{URL}.
Paper Structure (22 sections, 1 theorem, 21 equations, 13 figures, 10 tables, 2 algorithms)

This paper contains 22 sections, 1 theorem, 21 equations, 13 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Machine Unlearning
  4. Subspace Learning
  5. Preliminaries and Notation
  6. Proposed Method
  7. Find the Null Space for Unlearning
  8. Calibrate Decision Space
  9. Theoretical Analysis
  10. Null Space Prevents Over-unlearning
  11. Pseudo-labeling Benefits Unlearning
  12. Experiments
  13. Protocol and Evaluation Metrics
  14. Comparison with the State-of-the-Art
  15. Ablation Studies
  16. ...and 7 more sections

Key Result

Theorem 1

For a trained model $f(\cdot; \theta_o)$, unlearning the target samples from $\mathcal{D}_u$ in the null space tailored to the remaining samples from $\mathcal{D}_r$ ensures the unlearned model $f(\cdot; \theta_u)$ has approximately the same performance as $f(\cdot; \theta_o)$, i.e., where $\mathcal{L}_{r}(\theta) := \frac{1}{|\mathcal{D}_r|}\sum_{\mathcal{D}_r}\ell(f(\mathbf{x}_i;\theta),y_i)$ i

Figures (13)

  • Figure 2: Comparing the MIA accuracy of different unlearning methods. A higher $Acc_{MIA}$ indicates more effective unlearning.
  • Figure 3: (a) Distributions of pseudo-labels and predictions of the retrained model on CIFAR-10. (b) Loss contour on CIFAR-10. We evaluate the model's loss on remaining test data.
  • Figure 4: Visualization of latent space after unlearning all samples of class $3$ (red points) on CIFAR-10: (a) Original and (b) Unlearned model with all samples, (c) Original and (d) Unlearned models, both excluding the unlearned samples.
  • Figure 5: Comparison of class-wise accuracy on CIFAR-10. We compare the original model, the retrained model, and the model obtained by UNSC under different settings. In all experiments, the UNSC-obtained model demonstrates near-zero accuracy on the forgetting classes, achieving superior accuracy compared to the original model and comparable performance to the retrained model.
  • Figure 6: Class-wise accuracy of the original model, the retrained model, and the model obtained by UNSC on FashionMNIST. We present the results of unlearning different numbers of classes.
  • ...and 8 more figures

Theorems & Definitions (4)

  • Definition 1: Machine unlearning cao2015towards
  • Definition 2: Null space
  • Theorem 1
  • proof