Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SSSE: Efficiently Erasing Samples from Trained Machine Learning Models

Alexandra Peste, Dan Alistarh, Christoph H. Lampert

TL;DR

SSSE addresses the data-rights requirement to erase training samples from a trained model by deriving a closed-form, second-order update that uses the empirical Fisher Information Matrix in place of the Hessian. The update, $\hat{\theta}_{\epsilon} = \theta^\\star + \frac{\epsilon}{n-k} \hat{F}^{-1}_{\mathcal{D}}(\theta^\\star) \sum_{i \in \mathcal{S}} \nabla \ell_i(\theta^\\star)$, is computed efficiently via Sherman–Morrison updates and requires access only to the erased data, making it applicable to both convex and non-convex models. The authors introduce two evaluation schemes for multi-attribute binary and multinomial classification to measure how closely SSSE matches retraining from scratch and to guide hyper-parameter selection. Empirical results on CelebA, AwA2, and CIFAR-10 show that SSSE can erase samples nearly as effectively as retraining while offering substantial computational advantages, validating its practicality for real-world data deletion scenarios.

Abstract

The availability of large amounts of user-provided data has been key to the success of machine learning for many real-world tasks. Recently, an increasing awareness has emerged that users should be given more control about how their data is used. In particular, users should have the right to prohibit the use of their data for training machine learning systems, and to have it erased from already trained systems. While several sample erasure methods have been proposed, all of them have drawbacks which have prevented them from gaining widespread adoption. Most methods are either only applicable to very specific families of models, sacrifice too much of the original model's accuracy, or they have prohibitive memory or computational requirements. In this paper, we propose an efficient and effective algorithm, SSSE, for samples erasure, that is applicable to a wide class of machine learning models. From a second-order analysis of the model's loss landscape we derive a closed-form update step of the model parameters that only requires access to the data to be erased, not to the original training set. Experiments on three datasets, CelebFaces attributes (CelebA), Animals with Attributes 2 (AwA2) and CIFAR10, show that in certain cases SSSE can erase samples almost as well as the optimal, yet impractical, gold standard of training a new model from scratch with only the permitted data.

SSSE: Efficiently Erasing Samples from Trained Machine Learning Models

TL;DR

SSSE addresses the data-rights requirement to erase training samples from a trained model by deriving a closed-form, second-order update that uses the empirical Fisher Information Matrix in place of the Hessian. The update, , is computed efficiently via Sherman–Morrison updates and requires access only to the erased data, making it applicable to both convex and non-convex models. The authors introduce two evaluation schemes for multi-attribute binary and multinomial classification to measure how closely SSSE matches retraining from scratch and to guide hyper-parameter selection. Empirical results on CelebA, AwA2, and CIFAR-10 show that SSSE can erase samples nearly as effectively as retraining while offering substantial computational advantages, validating its practicality for real-world data deletion scenarios.

Abstract

The availability of large amounts of user-provided data has been key to the success of machine learning for many real-world tasks. Recently, an increasing awareness has emerged that users should be given more control about how their data is used. In particular, users should have the right to prohibit the use of their data for training machine learning systems, and to have it erased from already trained systems. While several sample erasure methods have been proposed, all of them have drawbacks which have prevented them from gaining widespread adoption. Most methods are either only applicable to very specific families of models, sacrifice too much of the original model's accuracy, or they have prohibitive memory or computational requirements. In this paper, we propose an efficient and effective algorithm, SSSE, for samples erasure, that is applicable to a wide class of machine learning models. From a second-order analysis of the model's loss landscape we derive a closed-form update step of the model parameters that only requires access to the data to be erased, not to the original training set. Experiments on three datasets, CelebFaces attributes (CelebA), Animals with Attributes 2 (AwA2) and CIFAR10, show that in certain cases SSSE can erase samples almost as well as the optimal, yet impractical, gold standard of training a new model from scratch with only the permitted data.

Paper Structure

This paper contains 22 sections, 1 theorem, 13 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Background
  5. Single-Step Sample Erasure
  6. Space and Computational Complexity.
  7. Illustrative Example.
  8. Analyzing the Approximation Between Hessian and Empirical FIM.
  9. Discussion.
  10. Evaluation Methods
  11. Multi-Attribute Binary Classification
  12. Multinomial Classification
  13. Discussion.
  14. Experiments
  15. Convex Multi-Attribute Classification
  16. ...and 7 more sections

Key Result

Proposition 1

Let $\theta \in \mathbb{R}^{c\times m}$ be the parameters of a linear model, with $m > c$ and $\mathop{\mathrm{rank}}\nolimits{\theta}=c$. For any training sample $x$, consider $p\in[0, 1]^c$ the output probabilities per class. Assume that exists $\epsilon\approx 0$, such that for any input label $y

Figures (7)

  • Figure 1: The decision boundary learned with SSSE and the influence function update versus the original model and retraining from scratch, when removing a single or multiple samples. We also differentiate between using the Fisher (or Hessian) on the LKO dataset instead of the full training set. The removed samples are represented by the larger transparent blue dots.
  • Figure 2: (CelebA) The similarity ratio $\gamma_{\mathcal{S}} (\hat{\theta}; \theta^\star, \theta^\star_{-\mathcal{S}})$, together with the normalized parameters distance for $\hat{\theta}_\epsilon$ on the removed samples $\mathcal{S}$, as a function of the scaling factor $\epsilon$. The results are averaged across multiple attributes, removed at different rates.
  • Figure 3: The normalized confusion distance. All samples removed in one step belong to a single class. We report the mean and standard deviation over the first 10 out of 50 classes for AwA2 and over the first 5 out of 10 classes for CIFAR10.
  • Figure 4: (AwA2) The absolute difference in accuracy between SSSE update $\hat{\theta}_\epsilon$ and $\theta^\star_{-\mathcal{S}}$, across all different data splits. Results are averaged across the first 10 different classes, for each considered removal percentage.
  • Figure 5: (CIFAR10) Average absolute difference in accuracy between $\hat{\theta}_\epsilon$ and $\theta^\star_{-\mathcal{S}}$, when using different batch sizes for $\hat{F}^{-1}_{\mathcal{D}}(\theta^\star)$ in SSSE. Using mini-batches of 10 gradients does not have a major negative impact on the accuracy of the resulting model. Average and standard deviations over the first 5 classes are reported.
  • ...and 2 more figures

Theorems & Definitions (7)

  • Definition 3.1: Single-Step Sample Erasure Update (SSSE)
  • Proposition 1
  • proof
  • Definition 4.1: Performance similarity
  • Definition 4.2: Similarity ratio between $\hat{\theta}_\epsilon$ and $\theta^\star$, $\theta^\star_{\mathcal{-S}}$
  • Definition 4.3: Confusion distance
  • Definition 4.4: Normalized confusion distance