Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Machine Unlearning by Model Splitting and Core Sample Selection

Maximilian Egger, Rawad Bitar, Rüdiger Urbanke

TL;DR

The paper tackles the challenge of efficiently and verifiably unlearning data from ML models under privacy regulations. It proposes MaxRR, a two-stage framework that combines unlearning-aware training with model splitting, replacing the final classifier with a linear SVM and training the feature extractor on a small core subset $\mathcal{D}_k$. This enables exact unlearning for many requests (especially when forgotten data lies outside the core) and practical approximate unlearning otherwise, paired with a verification pipeline based on confidence-based membership inference attacks. Empirical results on Fashion MNIST with LeNet-5 demonstrate that core-based training preserves performance while substantially reducing unlearning cost, and that MIA-based verification can validate unlearning in many scenarios, albeit with caveats. Overall, MaxRR provides a scalable, verifiable pathway to unlearning that can serve as a preprocessing step or integrate with stronger guarantees when needed.

Abstract

Machine unlearning is essential for meeting legal obligations such as the right to be forgotten, which requires the removal of specific data from machine learning models upon request. While several approaches to unlearning have been proposed, existing solutions often struggle with efficiency and, more critically, with the verification of unlearning - particularly in the case of weak unlearning guarantees, where verification remains an open challenge. We introduce a generalized variant of the standard unlearning metric that enables more efficient and precise unlearning strategies. We also present an unlearning-aware training procedure that, in many cases, allows for exact unlearning. We term our approach MaxRR. When exact unlearning is not feasible, MaxRR still supports efficient unlearning with properties closely matching those achieved through full retraining.

Efficient Machine Unlearning by Model Splitting and Core Sample Selection

TL;DR

The paper tackles the challenge of efficiently and verifiably unlearning data from ML models under privacy regulations. It proposes MaxRR, a two-stage framework that combines unlearning-aware training with model splitting, replacing the final classifier with a linear SVM and training the feature extractor on a small core subset . This enables exact unlearning for many requests (especially when forgotten data lies outside the core) and practical approximate unlearning otherwise, paired with a verification pipeline based on confidence-based membership inference attacks. Empirical results on Fashion MNIST with LeNet-5 demonstrate that core-based training preserves performance while substantially reducing unlearning cost, and that MIA-based verification can validate unlearning in many scenarios, albeit with caveats. Overall, MaxRR provides a scalable, verifiable pathway to unlearning that can serve as a preprocessing step or integrate with stronger guarantees when needed.

Abstract

Machine unlearning is essential for meeting legal obligations such as the right to be forgotten, which requires the removal of specific data from machine learning models upon request. While several approaches to unlearning have been proposed, existing solutions often struggle with efficiency and, more critically, with the verification of unlearning - particularly in the case of weak unlearning guarantees, where verification remains an open challenge. We introduce a generalized variant of the standard unlearning metric that enables more efficient and precise unlearning strategies. We also present an unlearning-aware training procedure that, in many cases, allows for exact unlearning. We term our approach MaxRR. When exact unlearning is not feasible, MaxRR still supports efficient unlearning with properties closely matching those achieved through full retraining.
Paper Structure (12 sections, 1 theorem, 4 equations, 6 figures, 2 tables, 2 algorithms)

This paper contains 12 sections, 1 theorem, 4 equations, 6 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. System Model and Preliminaries
  3. Illustrative Example of Efficient Unlearning
  4. Efficient Unlearning via Model Splitting and Core Sample Selection
  5. Unlearning-Aware Model Training
  6. Exact and Approximate Unlearning
  7. Exact Unlearning
  8. Approximate Unlearning
  9. Sensitivity Analysis for FE and SVM
  10. Verification of Unlearned Non-Core Samples
  11. Verification of Unlearning Smaller Fractions of Samples
  12. Model Architecture

Key Result

Proposition 1

Let $w_{\text{SVM}}$ be a linear SVM trained on the dataset $\mathcal{D}$, and let $\mathcal{S} \subset \mathcal{D}$ denote the resulting support vectors. For any $\mathcal{D}_f \subseteq \mathcal{D} \setminus \mathcal{S}$, retraining the SVM on $\mathcal{D} \setminus \mathcal{D}_f$ yields the same

Figures (6)

  • Figure 1: Unlearning-aware model architecture in MaxRR, $\mathcal{D}$ is the entire training set, $\mathcal{D}_{k}$ are the topmost $k$ important samples.
  • Figure 2: Comparison of empirical support vector frequency across 100 training runs.
  • Figure 3: Unlearning $\mathcal{D}_{20 \cdot 10^3}$ over $20$ runs $x \in \{1, \cdots, 20\}$.
  • Figure 4: Comparison of verification results for samples $\mathcal{D}_f$.
  • Figure 5: Unlearning random $20\cdot 10^3$ samples from $\mathcal{D}_{\lnot 20k}$
  • ...and 1 more figures

Theorems & Definitions (3)

  • Definition 1: Generalized exact unlearning
  • Proposition 1
  • proof