Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Certified Machine Unlearning via Noisy Stochastic Gradient Descent

Eli Chien, Haoyu Wang, Ziang Chen, Pan Li

TL;DR

This work introduces certified machine unlearning via projected noisy SGD (PNSGD) for convex objectives, establishing a first approximate unlearning guarantee by linking the initial $W_\infty$ distance between adjacent learning processes to the final Rényi divergence through contractive noisy iterations. The method yields explicit RU guarantees with bounds that depend on mini-batch size, step size, and data geometry, and demonstrates substantial computational savings over retraining while preserving utility under privacy constraints. The analysis scales to sequential and batch unlearning via $W_\infty$ tracking and triangle inequalities, and is complemented by experiments on MNIST and CIFAR-10 showing strong practicality with only a small fraction of gradient computations compared to baselines. Limitations include the strong convexity requirement, with future work potentially extending to non-convex settings using Langevin dynamics and broader unlearning scenarios.

Abstract

``The right to be forgotten'' ensured by laws for user data privacy becomes increasingly important. Machine unlearning aims to efficiently remove the effect of certain data points on the trained model parameters so that it can be approximately the same as if one retrains the model from scratch. We propose to leverage projected noisy stochastic gradient descent for unlearning and establish its first approximate unlearning guarantee under the convexity assumption. Our approach exhibits several benefits, including provable complexity saving compared to retraining, and supporting sequential and batch unlearning. Both of these benefits are closely related to our new results on the infinite Wasserstein distance tracking of the adjacent (un)learning processes. Extensive experiments show that our approach achieves a similar utility under the same privacy constraint while using $2\%$ and $10\%$ of the gradient computations compared with the state-of-the-art gradient-based approximate unlearning methods for mini-batch and full-batch settings, respectively.

Certified Machine Unlearning via Noisy Stochastic Gradient Descent

TL;DR

This work introduces certified machine unlearning via projected noisy SGD (PNSGD) for convex objectives, establishing a first approximate unlearning guarantee by linking the initial distance between adjacent learning processes to the final Rényi divergence through contractive noisy iterations. The method yields explicit RU guarantees with bounds that depend on mini-batch size, step size, and data geometry, and demonstrates substantial computational savings over retraining while preserving utility under privacy constraints. The analysis scales to sequential and batch unlearning via tracking and triangle inequalities, and is complemented by experiments on MNIST and CIFAR-10 showing strong practicality with only a small fraction of gradient computations compared to baselines. Limitations include the strong convexity requirement, with future work potentially extending to non-convex settings using Langevin dynamics and broader unlearning scenarios.

Abstract

``The right to be forgotten'' ensured by laws for user data privacy becomes increasingly important. Machine unlearning aims to efficiently remove the effect of certain data points on the trained model parameters so that it can be approximately the same as if one retrains the model from scratch. We propose to leverage projected noisy stochastic gradient descent for unlearning and establish its first approximate unlearning guarantee under the convexity assumption. Our approach exhibits several benefits, including provable complexity saving compared to retraining, and supporting sequential and batch unlearning. Both of these benefits are closely related to our new results on the infinite Wasserstein distance tracking of the adjacent (un)learning processes. Extensive experiments show that our approach achieves a similar utility under the same privacy constraint while using and of the gradient computations compared with the state-of-the-art gradient-based approximate unlearning methods for mini-batch and full-batch settings, respectively.
Paper Structure (31 sections, 29 theorems, 78 equations, 3 figures, 3 tables, 8 algorithms)

This paper contains 31 sections, 29 theorems, 78 equations, 3 figures, 3 tables, 8 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Converting Initial $W_\infty$ Distance to Final Rényi Divergence Bound
  5. Certified Unlearning Guaranatee for PNSGD
  6. Certified Unlearning Guarantees
  7. Unlearning Multiple Data Points
  8. Experiments
  9. Limitations and Conclusion
  10. Additional related works
  11. Limitations
  12. Broader Impact
  13. Standard definitions
  14. Existence of limiting distribution
  15. Proof of Theorem \ref{['thm:SGLD_unlearning_1']}
  16. ...and 16 more sections

Key Result

Lemma 2.6

Suppose $X_K\sim \text{CNI}_c(\mu_0,\{\psi_k\},\{\zeta_k\})$ and $X_K^\prime \sim \text{CNI}_c(\mu_0^\prime,\{\psi_k\},\{\zeta_k\})$ where the initial distribution satisfy $W_\infty(\mu_0,\mu_0^\prime)\leq Z$, the update function $\psi_k$ are $c$-contractive, and the noise distributions $\zeta_k = \

Figures (3)

  • Figure 1: The overview of PNSGD unlearning. (Left) Proof sketch for PNSGD unlearning guarantees. (Right) PNSGD (un)learning processes on adjacent datasets. Given a mini-batch sequence $\mathcal{B}$, the learning process $\mathcal{M}$ induces a regular polyhedron where each vertex corresponds to a stationary distribution $\nu_{\mathcal{D}|\mathcal{B}}$ for each dataset $\mathcal{D}$. $\nu_{\mathcal{D}|\mathcal{B}}$ and $\nu_{\mathcal{D|\mathcal{B}}^\prime}$ are adjacent if $\mathcal{D},\mathcal{D}^\prime$ differ in one data point. We provide an upper bound $Z_\mathcal{B}$ for the infinite Wasserstein distance $W_\infty(\nu_{\mathcal{D}|\mathcal{B}},\nu_{\mathcal{D|\mathcal{B}}^\prime})$, which is crucial for non-vacuous unlearning guarantees. Results of altschuler2022resolving allow us to convert the initial $W_\infty$ bound to Rényi difference bound $d_\alpha^{\mathcal{B}}$, and apply joint convexity of KL divergence to obtain the final privacy loss $\varepsilon$, which also take the randomness of $\mathcal{B}$ into account.
  • Figure 2: Illustration of (a) sequential unlearning and (b) batch unlearning. The key idea is to establish an upper bound on the initial $W_\infty$ distance. (a) For sequential unlearning, the initial $W_\infty$ distance bound $Z_\mathcal{B}^{(s)}$ for each $s^{th}$ unlearning request can be derived with triangle inequality. (b) For batch unlearning, we analyze the case that two learning processes can differ in $S\geq 1$ points.
  • Figure 3: Main experiments, where the top and bottom rows are for MNIST and CIFAR10 respectively. (a) Compare to baseline for unlearning one point using limited $K$ unlearning epoch. For PNSGD, we use only $K=1$ unlearning epoch. For D2D, we allow it to use $K=1,5$ unlearning epochs. (b) Unlearning $100$ points sequentially versus baseline. For LU, since their unlearning complexity only stays in a reasonable range when combined with batch unlearning of size $S$ sufficiently large, we report such a result only. (c,d) Noise-accuracy-complexity trade-off of PNSGD for unlearning $100$ points sequentially with various mini-batch sizes $b$, where all methods achieve $(\epsilon,1/n)$-unlearning guarantee with $\epsilon=0.01$. We also report the required accumulated epochs for retraining for each $b$.

Theorems & Definitions (45)

  • Definition 2.1: Rényi difference
  • Definition 2.2: Rényi Differential Privacy (RDP) mironov2017renyi
  • Definition 2.3: Rényi Unlearning (RU)
  • Definition 2.4: $W_\infty$ distance
  • Definition 2.5: Contractive Noisy Iteration ($c$-CNI)
  • Lemma 2.6: Metric-aware privacy amplification by iteration bound feldman2018privacy, simplified by altschuler2022resolving in Proposition 2.10
  • Theorem 3.1
  • Theorem 3.2: RU guarantee of PNSGD unlearning, fixed $\mathcal{B}$
  • Lemma 3.3: $W_\infty$ between adjacent PNSGD learning processes
  • Lemma 3.4: $W_\infty$ between PNSGD learning process to its stationary distribution
  • ...and 35 more