Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi
TL;DR
This work tackles efficient data deletion (unlearning) for convex models under adversarial, long update sequences by framing indistinguishability relative to full retraining. It introduces two gradient-based strategies: a simple Perturbed Gradient Descent (PGD) with Gaussian noise and a distributed, partitioned variant employing reservoir sampling and bootstrap sampling to maintain distributional properties. The authors provide rigorous ε-δ unlearning guarantees under strong convexity and smoothness, with weaker guarantees and secret-state alternatives for non-strongly convex cases, and demonstrate improved runtime-accuracy tradeoffs in high-dimensional settings. Through connections to differential privacy and distributed learning literature (e.g., SISA, ZDW12), the paper presents a principled framework for robust, privacy-preserving machine unlearning in convex ERM systems.
Abstract
We study the data deletion problem for convex models. By leveraging techniques from convex optimization and reservoir sampling, we give the first data deletion algorithms that are able to handle an arbitrarily long sequence of adversarial updates while promising both per-deletion run-time and steady-state error that do not grow with the length of the update sequence. We also introduce several new conceptual distinctions: for example, we can ask that after a deletion, the entire state maintained by the optimization algorithm is statistically indistinguishable from the state that would have resulted had we retrained, or we can ask for the weaker condition that only the observable output is statistically indistinguishable from the observable output that would have resulted from retraining. We are able to give more efficient deletion algorithms under this weaker deletion criterion.