Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Certified Unlearning in High Dimensions: A Hypothesis Testing Approach

Aaradhya Pandey, Arnab Auddy, Haolin Zou, Arian Maleki, Sanjeev Kulkarni

TL;DR

This paper tackles unlearning in high-dimensional settings where $p$ is comparable to $n$ and standard optimization assumptions fail. It introduces $\varepsilon$-Gaussian certifiability as a canonical, high-dimensional privacy notion and analyzes a one-step Newton unlearning procedure with Gaussian noise within a generalized linear model trained via Regularized ERM. The authors prove that a single Newton update plus calibrated Gaussian noise achieves $(\phi,\varepsilon)$-Gaussian certifiability and yields vanishing Generalization Error Divergence (GED) as $n,p \to \infty$ with $n/p \to \gamma$, provided the removal size $m$ satisfies $m = o(n^{1/4}/\mathrm{polylog}(n))$. Experiments on logistic/L2-regularized setups corroborate that Gaussian noise outperforms Laplace noise and that GED declines with dimension, supporting the theoretical guarantees. Overall, the work establishes a principled, scalable privacy-utility trade-off for unlearning in high dimensions and clarifies why Gaussian certifiability is optimal in this regime.

Abstract

Machine unlearning seeks to efficiently remove the influence of selected data while preserving generalization. Significant progress has been made in low dimensions $(p \ll n)$, but high dimensions pose serious theoretical challenges as standard optimization assumptions of $Ω(1)$ strong convexity and $O(1)$ smoothness of the per-example loss $f$ rarely hold simultaneously in proportional regimes $(p\sim n)$. In this work, we introduce $\varepsilon$-Gaussian certifiability, a canonical and robust notion well-suited to high-dimensional regimes, that optimally captures a broad class of noise adding mechanisms. Then we theoretically analyze the performance of a widely used unlearning algorithm based on one step of the Newton method in the high-dimensional setting described above. Our analysis shows that a single Newton step, followed by a well-calibrated Gaussian noise, is sufficient to achieve both privacy and accuracy in this setting. This result stands in sharp contrast to the only prior work that analyzes machine unlearning in high dimensions \citet{zou2025certified}, which relaxes some of the standard optimization assumptions for high-dimensional applicability, but operates under the notion of $\varepsilon$-certifiability. That work concludes %that a single Newton step is insufficient even for removing a single data point, and that at least two steps are required to ensure both privacy and accuracy. Our result leads us to conclude that the discrepancy in the number of steps arises because of the sub optimality of the notion of $\varepsilon$-certifiability and its incompatibility with noise adding mechanisms, which $\varepsilon$-Gaussian certifiability is able to overcome optimally.

Gaussian Certified Unlearning in High Dimensions: A Hypothesis Testing Approach

TL;DR

This paper tackles unlearning in high-dimensional settings where is comparable to and standard optimization assumptions fail. It introduces -Gaussian certifiability as a canonical, high-dimensional privacy notion and analyzes a one-step Newton unlearning procedure with Gaussian noise within a generalized linear model trained via Regularized ERM. The authors prove that a single Newton update plus calibrated Gaussian noise achieves -Gaussian certifiability and yields vanishing Generalization Error Divergence (GED) as with , provided the removal size satisfies . Experiments on logistic/L2-regularized setups corroborate that Gaussian noise outperforms Laplace noise and that GED declines with dimension, supporting the theoretical guarantees. Overall, the work establishes a principled, scalable privacy-utility trade-off for unlearning in high dimensions and clarifies why Gaussian certifiability is optimal in this regime.

Abstract

Machine unlearning seeks to efficiently remove the influence of selected data while preserving generalization. Significant progress has been made in low dimensions , but high dimensions pose serious theoretical challenges as standard optimization assumptions of strong convexity and smoothness of the per-example loss rarely hold simultaneously in proportional regimes . In this work, we introduce -Gaussian certifiability, a canonical and robust notion well-suited to high-dimensional regimes, that optimally captures a broad class of noise adding mechanisms. Then we theoretically analyze the performance of a widely used unlearning algorithm based on one step of the Newton method in the high-dimensional setting described above. Our analysis shows that a single Newton step, followed by a well-calibrated Gaussian noise, is sufficient to achieve both privacy and accuracy in this setting. This result stands in sharp contrast to the only prior work that analyzes machine unlearning in high dimensions \citet{zou2025certified}, which relaxes some of the standard optimization assumptions for high-dimensional applicability, but operates under the notion of -certifiability. That work concludes %that a single Newton step is insufficient even for removing a single data point, and that at least two steps are required to ensure both privacy and accuracy. Our result leads us to conclude that the discrepancy in the number of steps arises because of the sub optimality of the notion of -certifiability and its incompatibility with noise adding mechanisms, which -Gaussian certifiability is able to overcome optimally.
Paper Structure (38 sections, 8 theorems, 58 equations, 12 figures)

This paper contains 38 sections, 8 theorems, 58 equations, 12 figures.

Table of Contents

  1. Introduction and Theoretical contributions
  2. The machine unlearning problem: privacy and accuracy
  3. Generalized linear model (of data) and empirical risk minimization
  4. Formal definitions of certifiability and accuracy
  5. General certifiability through the lens of hypothesis testing
  6. (phi,ve)-Gaussian certfiability- a canonical choice in high dimensions
  7. Generalization Error Divergence: a measure of accuracy
  8. Newton-based machine unlearning: theoretical guarantees
  9. Newton-based unlearning mechanism
  10. Technical assumptions on the loss functions and the data
  11. Theoretical Guarantees: ensuring user privacy as well as model accuracy
  12. Experiments validating our theoretical contributions
  13. Conclusion
  14. Appendix of Gaussian certified unlearning in high dimensions: A Hypothesis Testing approach
  15. Motivations behind our approach on a high level
  16. ...and 23 more sections

Key Result

Lemma 1

For any $\bm{\mu}_1, \bm{\mu}_2 \in \mathbb{R}^p$ and $\sigma >0$ let $\varepsilon := \frac{1}{\sigma}\|\bm{\mu}_1- \bm{\mu}_2\|_2$. Then

Figures (12)

  • Figure 1: Comparison of unlearned estimators on new test data: mean GED (with 3 SD error bars) across the dimension $p$ (both in $\log$ scale) for Laplace (in red) vs. Gaussian (in cyan). We set $\lambda=0.5$.
  • Figure 2: Comparison on unlearned data: mean UED (with 3 SD error bars) on the unlearned set across $p$ (both in $\log$ scale) for Laplace (in red) vs. Gaussian noise (in cyan). We set $\lambda=0.5$.
  • Figure 3: Comparison of GED (plotted in $\log$ scale) across different values of $\varepsilon$ for Laplace noise (in red) vs. Gaussian noise (in cyan). We set $\lambda=0.5$.
  • Figure 4: Comparison of mean GED (with 3 SD error bars) across the unlearning size $m$ (both in $\log$ scale) for Laplace noise (in red) vs. Gaussian noise (in cyan). We set $\lambda=0.5$.
  • Figure 5: Comparison of the difference in negative log likelihood on the requested removal set (plotted in $\log$ scale) among the retrained estimator, with unlearned estimator with Laplace noise (in red) and Gaussian noise (in cyan). The left, middle, and right figures plot unlearning for removal sizes $m=1$, $m=5$, and $m=10$ respectively. Here $n=p=1255$.
  • ...and 7 more figures

Theorems & Definitions (13)

  • Definition 1
  • Definition 2: $f$-certifiability
  • Lemma 1: Dimension freeness
  • Definition 3
  • Definition 4
  • Theorem 2
  • Theorem 3
  • Definition 5
  • Theorem 4
  • Theorem 5
  • ...and 3 more