Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Sparsity Can Simplify Machine Unlearning

Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, Sijia Liu

TL;DR

This paper reframes machine unlearning by incorporating model sparsity through weight pruning, showing that sparse models reduce the gap between exact retraining and approximate unlearning. It introduces a prune-first-then-unlearn paradigm and a sparsity-aware unlearning framework that uses an L1 penalty, with extensive experiments across datasets and architectures demonstrating consistent gains in unlearning efficacy while preserving fidelity and generalization. The work also analyzes IU under ave-ERM vs sum-ERM, derives a bound linking sparsity to unlearning error, and demonstrates practical benefits in backdoor defense and transfer learning. Overall, sparsity emerges as a key lever to make MU faster and more reliable, enabling broader deployment under data removal regulations.

Abstract

In response to recent data regulation requirements, machine unlearning (MU) has emerged as a critical process to remove the influence of specific examples from a given model. Although exact unlearning can be achieved through complete model retraining using the remaining dataset, the associated computational costs have driven the development of efficient, approximate unlearning techniques. Moving beyond data-centric MU approaches, our study introduces a novel model-based perspective: model sparsification via weight pruning, which is capable of reducing the gap between exact unlearning and approximate unlearning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient. This leads to a new MU paradigm, termed prune first, then unlearn, which infuses a sparse model prior into the unlearning process. Building on this insight, we also develop a sparsity-aware unlearning method that utilizes sparsity regularization to enhance the training process of approximate unlearning. Extensive experiments show that our proposals consistently benefit MU in various unlearning scenarios. A notable highlight is the 77% unlearning efficacy gain of fine-tuning (one of the simplest unlearning methods) when using sparsity-aware unlearning. Furthermore, we demonstrate the practical impact of our proposed MU methods in addressing other machine learning challenges, such as defending against backdoor attacks and enhancing transfer learning. Codes are available at https://github.com/OPTML-Group/Unlearn-Sparse.

Model Sparsity Can Simplify Machine Unlearning

TL;DR

This paper reframes machine unlearning by incorporating model sparsity through weight pruning, showing that sparse models reduce the gap between exact retraining and approximate unlearning. It introduces a prune-first-then-unlearn paradigm and a sparsity-aware unlearning framework that uses an L1 penalty, with extensive experiments across datasets and architectures demonstrating consistent gains in unlearning efficacy while preserving fidelity and generalization. The work also analyzes IU under ave-ERM vs sum-ERM, derives a bound linking sparsity to unlearning error, and demonstrates practical benefits in backdoor defense and transfer learning. Overall, sparsity emerges as a key lever to make MU faster and more reliable, enabling broader deployment under data removal regulations.

Abstract

In response to recent data regulation requirements, machine unlearning (MU) has emerged as a critical process to remove the influence of specific examples from a given model. Although exact unlearning can be achieved through complete model retraining using the remaining dataset, the associated computational costs have driven the development of efficient, approximate unlearning techniques. Moving beyond data-centric MU approaches, our study introduces a novel model-based perspective: model sparsification via weight pruning, which is capable of reducing the gap between exact unlearning and approximate unlearning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient. This leads to a new MU paradigm, termed prune first, then unlearn, which infuses a sparse model prior into the unlearning process. Building on this insight, we also develop a sparsity-aware unlearning method that utilizes sparsity regularization to enhance the training process of approximate unlearning. Extensive experiments show that our proposals consistently benefit MU in various unlearning scenarios. A notable highlight is the 77% unlearning efficacy gain of fine-tuning (one of the simplest unlearning methods) when using sparsity-aware unlearning. Furthermore, we demonstrate the practical impact of our proposed MU methods in addressing other machine learning challenges, such as defending against backdoor attacks and enhancing transfer learning. Codes are available at https://github.com/OPTML-Group/Unlearn-Sparse.
Paper Structure (18 sections, 2 theorems, 17 equations, 8 figures, 13 tables)

This paper contains 18 sections, 2 theorems, 17 equations, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Revisiting Machine Unlearning and Evaluation
  3. Model Sparsity: A Missing Factor Influencing Machine Unlearning
  4. Sparsity-Aided Machine Unlearning
  5. Experiments
  6. Experiment setups
  7. Experiment results
  8. Related Work
  9. Conclusion
  10. Acknowledgement
  11. Proof of Proposition \ref{['prop: IU']}
  12. Proof of Proposition \ref{['prop: SGD_sparse_MU']}
  13. Additional Experimental Details and Results
  14. Datasets and models
  15. Additional training and unlearning settings
  16. ...and 3 more sections

Key Result

Proposition 1

Given the weighted ERM training $\boldsymbol{\theta}(\mathbf w) = \mathop{\mathrm{arg\,min}}\limits_{\boldsymbol{\theta}} L(\mathbf w,\boldsymbol{\theta})$ where $L(\mathbf w,\boldsymbol{\theta}) = \sum_{i=1}^N [w_i \ell_i (\boldsymbol{\theta}, \mathbf z_i)]$, $w_i \in [0,1]$ is the influence weight where $\mathbf 1$ is the $N$-dimensional vector of all ones, $\mathbf w=\mathbf{1}/N$ signifies the

Figures (8)

  • Figure 1: Schematic overview of our proposal on model sparsity-driven MU. Evaluation at-a-glance shows the performance of three unlearning methods (retraining-based exact unlearning, finetuning-based approximate unlearning golatkar2020eternal, and proposed unlearning on 95%-sparse model) under five metrics: unlearning accuracy (UA), membership inference attack (MIA)-based unlearning efficacy, accuracy on remaining data (RA), testing accuracy (TA), and run-time efficiency (RTE); see summary in Tab. \ref{['tab: summary_MU_methods_metrics']}. The unlearning scenario is given by class-wise forgetting, where data points of a single class are scrubbed. Each metric is normalized to $[0,1]$ based on the best result across unlearning methods for ease of visualization. Results indicate that model sparsity reduces the gap between exact and approximate MU without loss in efficiency.
  • Figure 2: Testing accuracy of OMP-based sparse ResNet-18 vs. the dense model on CIFAR-10.
  • Figure 3: Performance of approximate unlearning (FT, GA, FF, IU) and exact unlearning (Retrain) in efficacy (UA and MIA-Efficacy), fidelity (RA), and generalization (TA) vs. model sparsity (achieved by OMP) in the data-model setup (CIFAR-10, ResNet-18). The unlearning scenario is class-wise forgetting, and the average unlearning performance over 10 classes is reported. We remark that being closer to Retrain performance is better for approximate MU schemes.
  • Figure 4: Influence of different pruning methods (SynFlow, OMP, and IMP) in unlearning efficacy (UA and MIA-Efficacy) and generalization (TA) on (CIFAR-10, ResNet-18). Left: UA vs. TA. Right: MIA-Efficacy vs. TA. Each point is a FT-based unlearned dense or sparse model (75% or 95% sparsity), or a retrained dense model.
  • Figure 5: Performance of sparsity-aware unlearning vs. FT and Retrain on class-wise forgetting and random data forgetting under (CIFAR-10, ResNet-18). Each metric is normalized to $[0,1]$ based on the best result across unlearning methods for ease of visualization, while the actual best value is provided (e.g., $2.52$ is the least computation time for class-wise forgetting).
  • ...and 3 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2