Descend or Rewind? Stochastic Gradient Descent Unlearning
Siqiao Mu, Diego Klabjan
TL;DR
The paper addresses certified unlearning for stochastic gradient methods by recasting unlearning as biased or disturbed gradient dynamics. It develops a coupling-based framework and introduces a relaxed Gaussian mechanism to prove $(\epsilon,\delta)$ guarantees for three SGD-based unlearning variants: PSGD-R2D, SGD-R2D, and SGD-D2D, across strongly convex, convex, and nonconvex losses. Key insights show D2D offers tighter guarantees for strongly convex problems, while R2D provides universal computational advantages by rewinding to earlier states and can achieve unlearning in nonconvex settings. The results establish practical, black-box unlearning with end-of-training noise injection, enabling scalable and privacy-preserving data removal in large-scale models, with clear directions for extending to generalization guarantees and experimental validation.
Abstract
Machine unlearning algorithms aim to remove the impact of selected training data from a model without the computational expenses of retraining from scratch. Two such algorithms are ``Descent-to-Delete" (D2D) and ``Rewind-to-Delete" (R2D), full-batch gradient descent algorithms that are easy to implement and satisfy provable unlearning guarantees. In particular, the stochastic version of D2D is widely implemented as the ``finetuning" unlearning baseline, despite lacking theoretical backing on nonconvex functions. In this work, we prove $(ε, δ)$ certified unlearning guarantees for stochastic R2D and D2D for strongly convex, convex, and nonconvex loss functions, by analyzing unlearning through the lens of disturbed or biased gradient systems, which may be contracting, semi-contracting, or expansive respectively. Our argument relies on optimally coupling the random behavior of the unlearning and retraining trajectories, resulting in a probabilistic sensitivity bound that can be combined with a novel relaxed Gaussian mechanism to achieve $(ε, δ)$ unlearning. We determine that D2D can yield tighter guarantees for strongly convex functions compared to R2D by relying on contraction to a unique global minimum. However, unlike D2D, R2D can achieve unlearning in the convex and nonconvex setting because it draws the unlearned model closer to the retrained model by reversing the accumulated disturbances.