Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate Data Deletion from Machine Learning Models

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, James Zou

TL;DR

This work tackles approximate data deletion for trained ML models under privacy and regulatory pressures, proposing the Projective Residual Update (PRU) that achieves deletion with time complexity $O(k^2 d)$, i.e., linear in the feature dimension and independent of dataset size $n$. PRU works by projecting the exact leave-$k$-out update onto the span of the $k$ deleted feature vectors, while leveraging offline hat-matrix and low-rank decompositions to keep computations efficient. The authors also introduce the Feature Injection Test (FIT) to measure how thoroughly a deletion method removes knowledge of a highly predictive feature, highlighting privacy-sensitive performance beyond traditional parameter proximity measures. Empirical results on synthetic data and real-world Yelp data show substantial runtime gains over exact retraining, with PRU often matching or approaching Newton-based deletion in accuracy, especially in high-dimensional or sparse settings, and FIT revealing privacy advantages not captured by L2 distance alone.

Abstract

Deleting data from a trained machine learning (ML) model is a critical task in many applications. For example, we may want to remove the influence of training points that might be out of date or outliers. Regulations such as EU's General Data Protection Regulation also stipulate that individuals can request to have their data deleted. The naive approach to data deletion is to retrain the ML model on the remaining data, but this is too time consuming. In this work, we propose a new approximate deletion method for linear and logistic models whose computational cost is linear in the the feature dimension $d$ and independent of the number of training data $n$. This is a significant gain over all existing methods, which all have superlinear time dependence on the dimension. We also develop a new feature-injection test to evaluate the thoroughness of data deletion from ML models.

Approximate Data Deletion from Machine Learning Models

TL;DR

This work tackles approximate data deletion for trained ML models under privacy and regulatory pressures, proposing the Projective Residual Update (PRU) that achieves deletion with time complexity , i.e., linear in the feature dimension and independent of dataset size . PRU works by projecting the exact leave--out update onto the span of the deleted feature vectors, while leveraging offline hat-matrix and low-rank decompositions to keep computations efficient. The authors also introduce the Feature Injection Test (FIT) to measure how thoroughly a deletion method removes knowledge of a highly predictive feature, highlighting privacy-sensitive performance beyond traditional parameter proximity measures. Empirical results on synthetic data and real-world Yelp data show substantial runtime gains over exact retraining, with PRU often matching or approaching Newton-based deletion in accuracy, especially in high-dimensional or sparse settings, and FIT revealing privacy advantages not captured by L2 distance alone.

Abstract

Deleting data from a trained machine learning (ML) model is a critical task in many applications. For example, we may want to remove the influence of training points that might be out of date or outliers. Regulations such as EU's General Data Protection Regulation also stipulate that individuals can request to have their data deleted. The naive approach to data deletion is to retrain the ML model on the remaining data, but this is too time consuming. In this work, we propose a new approximate deletion method for linear and logistic models whose computational cost is linear in the the feature dimension and independent of the number of training data . This is a significant gain over all existing methods, which all have superlinear time dependence on the dimension. We also develop a new feature-injection test to evaluate the thoroughness of data deletion from ML models.

Paper Structure

This paper contains 30 sections, 6 theorems, 42 equations, 1 figure, 10 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Notation and Problem Setup
  3. Methods
  4. Exact retraining
  5. Newton's method
  6. Influence method
  7. The Projective Residual Update
  8. The LKO, PseudoInv, and FastMult subroutines
  9. Outlier deletion
  10. Extension to logistic regression
  11. Evaluation Metrics
  12. L2 distance
  13. Feature injection test
  14. Empirical Validation
  15. Linear regression
  16. ...and 15 more sections

Key Result

Theorem 1

Algorithm alg:res computes $\theta^{\mathrm{res}}{} = \theta^{\mathrm{full}}{} + \textrm{proj}_{\textrm{span}(x_1,\ldots,x_k)}(\theta^{\setminus k}{} - \theta^{\mathrm{full}}{})$ with computational cost $O(k^2d)$.

Figures (1)

  • Figure 1: Yelp FIT experiment. We plot the mean of each metric $\pm$ the standard error of the mean. The PRU deletes the injected feature much more effectively and exhibits greater stability.

Theorems & Definitions (13)

  • Theorem 1
  • Definition 2
  • Corollary 3
  • proof
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • proof : Proof of Theorem \ref{['main']}
  • Proposition 7
  • proof
  • ...and 3 more