Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Machine Unlearning in Low-Dimensional Feature Subspace

Kun Fang, Qinghua Tao, Junxu Liu, Yaxin Xiao, Qingqing Ye, Jian Sun, Haibo Hu

TL;DR

The paper tackles the challenge of machine unlearning (MU) by introducing a low-dimensional feature subspace perspective to remove the influence of forgetting data while preserving remaining-data performance.It proposes LOFT, a PCA-inspired method that learns a projection matrix ${\bf U}$ on the feature covariances of the pretrained model, and plugs a small subspace module into the model to achieve unlearning without retraining or accessing raw data repeatedly.LOFT optimizes a joint objective on two covariance matrices to minimize forgetting information ($J_{\rm fg}$) while preserving remaining information ($J_{\rm rm}$), with optimization on the Stiefel manifold and a one-shot covariance computation.Extensive experiments across multiple datasets, models, and unlearning scenarios demonstrate that LOFT achieves superior unlearning accuracy and efficiency, while offering a practical, privacy-friendly plug-in deployment for scalable real-world MU.

Abstract

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model skithrough principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. In training, LOFT simply optimizes a small-size projection matrix flexibly plugged into the pretrained model, and only requires one-shot feature fetching from the pretrained backbone instead of repetitively accessing the raw data. Hence, LOFT mitigates two critical issues in mainstream MU methods, i.e., the privacy leakage risk from massive data reload and the inefficiency of updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is anonymously available at https://anonymous.4open.science/r/4352/.

Machine Unlearning in Low-Dimensional Feature Subspace

TL;DR

The paper tackles the challenge of machine unlearning (MU) by introducing a low-dimensional feature subspace perspective to remove the influence of forgetting data while preserving remaining-data performance.It proposes LOFT, a PCA-inspired method that learns a projection matrix ${\bf U}$ on the feature covariances of the pretrained model, and plugs a small subspace module into the model to achieve unlearning without retraining or accessing raw data repeatedly.LOFT optimizes a joint objective on two covariance matrices to minimize forgetting information ($J_{\rm fg}$) while preserving remaining information ($J_{\rm rm}$), with optimization on the Stiefel manifold and a one-shot covariance computation.Extensive experiments across multiple datasets, models, and unlearning scenarios demonstrate that LOFT achieves superior unlearning accuracy and efficiency, while offering a practical, privacy-friendly plug-in deployment for scalable real-world MU.

Abstract

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model skithrough principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. In training, LOFT simply optimizes a small-size projection matrix flexibly plugged into the pretrained model, and only requires one-shot feature fetching from the pretrained backbone instead of repetitively accessing the raw data. Hence, LOFT mitigates two critical issues in mainstream MU methods, i.e., the privacy leakage risk from massive data reload and the inefficiency of updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is anonymously available at https://anonymous.4open.science/r/4352/.
Paper Structure (61 sections, 3 theorems, 13 equations, 3 figures, 17 tables)

This paper contains 61 sections, 3 theorems, 13 equations, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Machine unlearning.
  4. Principal component analysis.
  5. Notations.
  6. Feature Subspace Perspective for Approximate MU
  7. Spectrum analysis.
  8. Reconstruction analysis.
  9. Analytical evidence.
  10. Approximate MU with Subspace Learning
  11. Optimization
  12. Implementation and Discussion
  13. One-shot feature fetching.
  14. Plug-in module implementation.
  15. Theoretical guarantee.
  16. ...and 46 more sections

Key Result

Lemma 3.1

Given the feature covariance ${\bf\Sigma}_{\rm rm}^{\rm exact}\xspace$ from $f_{\rm exact}\xspace$ on $\mathcal{D}_{\rm rm}\xspace$, its eigendecomposition is given by ${\bf\Sigma}_{\rm rm}^{\rm exact}\xspace={\bf V}{\bf\Lambda}{\bf V}^\top$ with eigenvalues $\lambda_1\geq\lambda_2\geq\cdots\lambda_

Figures (3)

  • Figure 1: Overview of our proposed LOFT and mainstream approximate MU methods. After unlearning, the trained projection matrix of LOFT is incorporated into the model's forward propagation during inference, with details in Sec.\ref{['sec:sun:implementation']}.
  • Figure 2: Spectrum and reconstruction analyses on features of $\mathcal{D}_{\rm fg}\xspace$ and $\mathcal{D}_{\rm rm}\xspace$ w.r.t. the pretrained $f_{\rm pre}\xspace$ and the retrained $f_{\rm exact}\xspace$, with results on the top-12 normalized eigenvalues and reconstruction errors.
  • Figure 3: A sensitivity analysis on (i) subspace dimensions $s$ ( left) and (ii) different positions of the projection matrix $\bf U$ ( right).

Theorems & Definitions (6)

  • Lemma 3.1
  • Proposition 4.1
  • Theorem 4.2
  • proof
  • Remark 2.1
  • proof