Certified Per-Instance Unlearning Using Individual Sensitivity Bounds

TL;DR

The paper tackles certified machine unlearning by moving from uniform worst-case noise to per-instance noise calibrated to the erased point’s influence. It builds a learn–then–unlearn framework using Langevin dynamics and Gaussian differential privacy, deriving high-probability, per-instance sensitivity bounds for ridge regression and showing how to tune unlearning noise accordingly. The main contributions are a trajectory-based GDP accounting method under contractive updates, and explicit per-instance unlearning guarantees for ridge regression, supported by experiments in MNIST and CIFAR-10 that reveal substantial heterogeneity across data points. The work demonstrates improved privacy–utility trade-offs through instance-aware calibration and motivates extending per-instance guarantees to more complex, non-linear models. Overall, it advances practical, provable unlearning by leveraging data-point specific influence, contraction properties, and GDP-based privacy accounting.

Abstract

Certified machine unlearning can be achieved via noise injection leading to differential privacy guarantees, where noise is calibrated to worst-case sensitivity. Such conservative calibration often results in performance degradation, limiting practical applicability. In this work, we investigate an alternative approach based on adaptive per-instance noise calibration tailored to the individual contribution of each data point to the learned solution. This raises the following challenge: how can one establish formal unlearning guarantees when the mechanism depends on the specific point to be removed? To define individual data point sensitivities in noisy gradient dynamics, we consider the use of per-instance differential privacy. For ridge regression trained via Langevin dynamics, we derive high-probability per-instance sensitivity bounds, yielding certified unlearning with substantially less noise injection. We corroborate our theoretical findings through experiments in linear settings and provide further empirical evidence on the relevance of the approach in deep learning settings.

Figures (11)

• Figure 1: Intuition behind per-instance unlearning. Top: After training, the parameter distributions induced by $\mathcal{A}(D)$ and $\mathcal{A}(D^{-i})$ are sharply concentrated and well separated, resulting in a large privacy gap $\varepsilon_0$. Bottom: After applying the unlearning procedure $\mathcal{U}$, the resulting distributions corresponding to $\mathcal{U}(\mathcal{A}(D),D^{-i},(x_i,y_i))$ and $\mathcal{U}(\mathcal{A}(D^{-i}),D^{-i},\varnothing)$ become closer, yielding a much smaller privacy parameter $\varepsilon \ll \varepsilon_0$.
• Figure 2: Learn--unlearn vs. retraining trajectories. The solid path $(\theta_k)$ corresponds to training on the full dataset $D$ followed by unlearning, while the dashed path $(\theta'_k)$ corresponds to (fictitiously) training and retraining on the retain dataset $D^{-i}$.
• Figure 3: High-probability sensitivity bounds $s_{i,k}^{\delta_{\mathrm s}}$ for all MNIST training points in the linear ridge regression setting. Rows correspond to data points and columns to learning iterations. Points are sorted by increasing final sensitivity $s_{i,T}^{\delta_{\mathrm s}}$, highlighting substantial per-instance heterogeneity already at the level of analytical bounds.
• Figure 4: Representative MNIST examples selected for unlearning, ordered from left to right by increasing difficulty, as measured by the per-sample gradient norm $\|\nabla_\theta \ell(\theta_T; x_i, y_i)\|$ at convergence for a certain seed.
• Figure 5: Privacy--utility trade-off on MNIST. Final test accuracy as a function of the privacy budget $\varepsilon$ for per-instance unlearning (solid lines) and a uniform baseline (dashed lines). Each curve corresponds to a distinct removed training point, highlighting the heterogeneity of the trade-off across points.

Theorems & Definitions (17)

• Definition 3.1: Differential Privacy dwork2006tcc
• Definition 3.2: $(\varepsilon,\delta)$-Reference Unlearning ginart2019
• Definition 3.3: $(\varepsilon,\delta)$-Unlearning sekhari2021
• Definition 3.4: $(\varepsilon,\delta)$-Per-Instance Unlearning
• Proposition 4.2: Gaussian DP accounting of the unlearning mechanism $\mathcal{U}$ for bounded per-instance sensitivities
• Theorem 4.3: Certified per-instance unlearning for ridge regression
• Proposition 4.4: High-probability sensitivity bounds $s_{i,k}^{\delta_{\mathrm s}}$ for ridge regression
• Remark 4.5: Exact unlearning for ridge regression
• Definition A.1: Gaussian Differential Privacy dong2019gdp
• Lemma B.1: Shifted step lemma (bok2024shiftedinterpolationdifferentialprivacy)