Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks

Odelia Melamed, Gilad Yehudai, Gal Vardi

TL;DR

The paper studies provable unlearning by applying a single gradient ascent step of size $\beta$ to the training loss on the forgotten sample, leveraging a KKT-based margin framework. It proves that this gradient-ascent unlearning is $(\epsilon,\delta,\tau)$-successful for both linear predictors and two-layer ReLU networks under high-dimensional, near-orthogonal data, with explicit dependence on data parameters such as $\psi$, $\phi$, and $\epsilon_d$. Moreover, the unlearned model remains close to the retrained max-margin solution on the retained data and preserves generalization on synthetic Gaussian-mixture distributions. Together, these results provide a rigorous, computationally efficient pathway to forgetting without full retraining, linking implicit bias, margin theory, and post-training updates to practical privacy-preserving objectives.

Abstract

Machine Unlearning aims to remove specific data from trained models, addressing growing privacy and ethical concerns. We provide a theoretical analysis of a simple and widely used method - gradient ascent - used to reverse the influence of a specific data point without retraining from scratch. Leveraging the implicit bias of gradient descent towards solutions that satisfy the Karush-Kuhn-Tucker (KKT) conditions of a margin maximization problem, we quantify the quality of the unlearned model by evaluating how well it satisfies these conditions w.r.t. the retained data. To formalize this idea, we propose a new success criterion, termed \textbf{$(ε, δ, τ)$-successful} unlearning, and show that, for both linear models and two-layer neural networks with high dimensional data, a properly scaled gradient-ascent step satisfies this criterion and yields a model that closely approximates the retrained solution on the retained data. We also show that gradient ascent performs successful unlearning while still preserving generalization in a synthetic Gaussian-mixture setting.

Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks

TL;DR

The paper studies provable unlearning by applying a single gradient ascent step of size to the training loss on the forgotten sample, leveraging a KKT-based margin framework. It proves that this gradient-ascent unlearning is -successful for both linear predictors and two-layer ReLU networks under high-dimensional, near-orthogonal data, with explicit dependence on data parameters such as , , and . Moreover, the unlearned model remains close to the retrained max-margin solution on the retained data and preserves generalization on synthetic Gaussian-mixture distributions. Together, these results provide a rigorous, computationally efficient pathway to forgetting without full retraining, linking implicit bias, margin theory, and post-training updates to practical privacy-preserving objectives.

Abstract

Machine Unlearning aims to remove specific data from trained models, addressing growing privacy and ethical concerns. We provide a theoretical analysis of a simple and widely used method - gradient ascent - used to reverse the influence of a specific data point without retraining from scratch. Leveraging the implicit bias of gradient descent towards solutions that satisfy the Karush-Kuhn-Tucker (KKT) conditions of a margin maximization problem, we quantify the quality of the unlearned model by evaluating how well it satisfies these conditions w.r.t. the retained data. To formalize this idea, we propose a new success criterion, termed \textbf{-successful} unlearning, and show that, for both linear models and two-layer neural networks with high dimensional data, a properly scaled gradient-ascent step satisfies this criterion and yields a model that closely approximates the retrained solution on the retained data. We also show that gradient ascent performs successful unlearning while still preserving generalization in a synthetic Gaussian-mixture setting.
Paper Structure (40 sections, 32 theorems, 279 equations, 1 figure)

This paper contains 40 sections, 32 theorems, 279 equations, 1 figure.

Table of Contents

  1. Introduction
  2. implicit bias and margin maximization
  3. Settings
  4. Notations.
  5. Architectures and training
  6. An objective for unlearning
  7. Unlearning with gradient ascent
  8. Data
  9. Linear Predictors
  10. Two-Layer ReLU Networks
  11. Proof sketch
  12. Unlearning batches of data points
  13. Generalization of the Unlearned Classifier
  14. Discussion and future work
  15. Proofs of data preliminaries for section \ref{['sec:settings']}
  16. ...and 25 more sections

Key Result

Theorem 2.1

Let $N(\mathbf{x},{\boldsymbol{\theta}})$ be a homogeneous linear or ReLU neural network. Consider minimizing the logistic or exponential loss using gradient flow over a binary classification set $S=\{(\mathbf{x}_i,y_i)\}_{i=1}^m \subseteq {\mathbb R}^d \times \{-1, 1\}$. Assume that there is a time

Figures (1)

  • Figure 1: Effect of deviation from the correct step size on the KKT approximation parameter $\epsilon$ for a two-layer network. The $x$-axis shows the step size as a fraction of the step size from Theorem \ref{['thm:nonlinunlearning']}, and the $y$-axis shows the KKT approximation parameter $\epsilon$ of the unlearned model w.r.t. the retain dataset.

Theorems & Definitions (38)

  • Theorem 2.1: lyu2019gradient, ji2020directional
  • Definition 2.1
  • Theorem 2.2: lyu2019gradient, ji2020directional
  • Definition 2.2: successful unlearning
  • Example
  • Theorem 3.1
  • Theorem 4.1
  • Theorem 6.1
  • Theorem A.1
  • Lemma A.1
  • ...and 28 more