Certified Data Removal Under High-dimensional Settings

TL;DR

This work studies certifiable data removal in high-dimensional settings where $n/p \to \gamma_0\in(0,\infty)$. It proposes a Perturbed Newton unlearning algorithm that starts from the full-data estimator, executes $T$ Newton steps, and adds isotropic Laplacian noise to achieve probabilistic certifiability (PAR). The authors show that in PHAS a single Newton step is insufficient for reliable unlearning, but two steps suffice for $m=O(1)$, with explicit bounds on perturbation scales and the GED accuracy measure, and they provide a detailed trade-off analysis between certifiability and accuracy. The results are supported by theoretical theorems under PHAS and extensive numerical experiments illustrating the benefits of the two-step approach and the role of noise in certifiability. Overall, the paper delivers a principled, high-dimensional framework for certifiable data removal with concrete guidance on when multiple Newton iterations are required and how to balance privacy and predictive utility in practice.

Abstract

Machine unlearning focuses on the computationally efficient removal of specific training data from trained models, ensuring that the influence of forgotten data is effectively eliminated without the need for full retraining. Despite advances in low-dimensional settings, where the number of parameters \( p \) is much smaller than the sample size \( n \), extending similar theoretical guarantees to high-dimensional regimes remains challenging. We propose an unlearning algorithm that starts from the original model parameters and performs a theory-guided sequence of Newton steps \( T \in \{ 1,2\}\). After this update, carefully scaled isotropic Laplacian noise is added to the estimate to ensure that any (potential) residual influence of forget data is completely removed. We show that when both \( n, p \to \infty \) with a fixed ratio \( n/p \), significant theoretical and computational obstacles arise due to the interplay between the complexity of the model and the finite signal-to-noise ratio. Finally, we show that, unlike in low-dimensional settings, a single Newton step is insufficient for effective unlearning in high-dimensional problems -- however, two steps are enough to achieve the desired certifiebility. We provide numerical experiments to support the certifiability and accuracy claims of this approach.

Figures (4)

• Figure 1: The impact of $T$ Newton iterations on the accuracy of certified data removal for $T=1,2$. On both plots, the $X$ axis denotes the exact leave-one-out loss. Then, the figure on the left plots the one Newton step plus noise loss on the $Y$ axis. The figure on the right plots the two Newton step plus noise loss on the $Y$ axis.
• Figure 2: The approximation and exact removal error for ridge logistic regression as function of $p$. To compare the one and two Newton step performances, no noise is added, leading to non-certified machine unlearning. Left: The one Newton step approximation error $\| \tilde{\bm \beta}_{ \backslash \mathcal{M}}^{(1)} - \bm{ \hat{\beta}}_{\backslash \mathcal{M} } \|_2$. Middle: The two Newton step approximation error $\|\tilde{\bm \beta}_{ \backslash \mathcal{M}}^{(2)} - \bm{ \hat{\beta}}_{\backslash \mathcal{M} } \|_2$. Right: The exact removal error $\| \bm{ \hat{\beta}}_{\backslash \mathcal{M} } - \bm{\hat{\beta}} \|_2$.
• Figure 3: The approximation and exact removal error error for ridge logistic regression as function of $m=|\mathcal{M}|$. Left: The one Newton step approximation error $\| \tilde{\bm \beta}_{ \backslash \mathcal{M}}^{(1)} - \bm{ \hat{\beta}}_{\backslash \mathcal{M} } \|_2$. Middle: The two Newton step approximation error $\|\tilde{\bm \beta}_{ \backslash \mathcal{M}}^{(2)} - \bm{ \hat{\beta}}_{\backslash \mathcal{M} } \|_2$. Right: The exact removal error $\| \bm{ \hat{\beta}}_{\backslash \mathcal{M} } - \bm{\hat{\beta}} \|_2$.
• Figure 4: Comparisons between the loss of the exactly unlearned models and its first and second Newton step approximations for ridge logistic regression as function of $p$. Top row: Tested on the left out data point. Left: loss of the exactly unlearned model. Middle: absolute error between the exactly unlearned model and its one Newton approximation. Right: absolute error between the exactly unlearned model and its second Newton approximation. Buttom row: Tested on an unseen data point. Left: loss of the exactly unlearned model. Middle: absolute error between the exactly unlearned model and its one Newton approximation. Right: absolute error between the exactly unlearned model and its second Newton approximation.

Theorems & Definitions (49)

• Definition 2.1: $(\phi,\epsilon)$- Probabilistically certified approximate data removal (PAR)
• Remark 2.1.1
• Remark 2.1.2
• Remark 2.1.3
• Definition 2.2: Generalization Error Divengence (GED)
• Definition 2.3: Proportional High-dimensional Asymptotic Setting (PHAS)
• Definition 2.4: Newton Method
• Example 3.1: Linear regression
• Example 3.2: Logistic regression
• Theorem 3.1