Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Some Models Resist Unlearning: A Linear Stability Perspective

Wei-Kai Chang, Rajiv Khanna

TL;DR

This work reframes machine unlearning as an asymptotic linear stability problem around a pre-trained minimum $w^*$. By introducing data coherence and decomposing it into retain/forget components, the authors derive explicit thresholds on $\lambda_{\max}(D)$ and the coherence $\sigma$ that separate stable forgetting from instability, tying the geometry of data to unlearning dynamics. A two-layer ReLU CNN data model shows that lower SNR (greater memorization) reduces coherence and facilitates forgetting, while higher SNR increases coherence and resists forgetting; these predictions are supported by Hessian-based diagnostics and heatmaps. The results yield a first principled account of the memorization–forgetting trade-off in unlearning, with practical implications for designing unlearning procedures and diagnostics in modern networks using random-matrix theory and stability analysis.

Abstract

Machine unlearning, the ability to erase the effect of specific training samples without retraining from scratch, is critical for privacy, regulation, and efficiency. However, most progress in unlearning has been empirical, with little theoretical understanding of when and why unlearning works. We tackle this gap by framing unlearning through the lens of asymptotic linear stability to capture the interaction between optimization dynamics and data geometry. The key quantity in our analysis is data coherence which is the cross sample alignment of loss surface directions near the optimum. We decompose coherence along three axes: within the retain set, within the forget set, and between them, and prove tight stability thresholds that separate convergence from divergence. To further link data properties to forgettability, we study a two layer ReLU CNN under a signal plus noise model and show that stronger memorization makes forgetting easier: when the signal to noise ratio (SNR) is lower, cross sample alignment is weaker, reducing coherence and making unlearning easier; conversely, high SNR, highly aligned models resist unlearning. For empirical verification, we show that Hessian tests and CNN heatmaps align closely with the predicted boundary, mapping the stability frontier of gradient based unlearning as a function of batching, mixing, and data/model alignment. Our analysis is grounded in random matrix theory tools and provides the first principled account of the trade offs between memorization, coherence, and unlearning.

Why Some Models Resist Unlearning: A Linear Stability Perspective

TL;DR

This work reframes machine unlearning as an asymptotic linear stability problem around a pre-trained minimum . By introducing data coherence and decomposing it into retain/forget components, the authors derive explicit thresholds on and the coherence that separate stable forgetting from instability, tying the geometry of data to unlearning dynamics. A two-layer ReLU CNN data model shows that lower SNR (greater memorization) reduces coherence and facilitates forgetting, while higher SNR increases coherence and resists forgetting; these predictions are supported by Hessian-based diagnostics and heatmaps. The results yield a first principled account of the memorization–forgetting trade-off in unlearning, with practical implications for designing unlearning procedures and diagnostics in modern networks using random-matrix theory and stability analysis.

Abstract

Machine unlearning, the ability to erase the effect of specific training samples without retraining from scratch, is critical for privacy, regulation, and efficiency. However, most progress in unlearning has been empirical, with little theoretical understanding of when and why unlearning works. We tackle this gap by framing unlearning through the lens of asymptotic linear stability to capture the interaction between optimization dynamics and data geometry. The key quantity in our analysis is data coherence which is the cross sample alignment of loss surface directions near the optimum. We decompose coherence along three axes: within the retain set, within the forget set, and between them, and prove tight stability thresholds that separate convergence from divergence. To further link data properties to forgettability, we study a two layer ReLU CNN under a signal plus noise model and show that stronger memorization makes forgetting easier: when the signal to noise ratio (SNR) is lower, cross sample alignment is weaker, reducing coherence and making unlearning easier; conversely, high SNR, highly aligned models resist unlearning. For empirical verification, we show that Hessian tests and CNN heatmaps align closely with the predicted boundary, mapping the stability frontier of gradient based unlearning as a function of batching, mixing, and data/model alignment. Our analysis is grounded in random matrix theory tools and provides the first principled account of the trade offs between memorization, coherence, and unlearning.
Paper Structure (25 sections, 14 theorems, 76 equations, 2 figures, 1 table)

This paper contains 25 sections, 14 theorems, 76 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theory
  4. Background
  5. Coherence Measures for Unlearning
  6. Coherence in single-dataset SGD.
  7. Coherence with retain and forget sets.
  8. Linear Stability Analysis of Unlearning
  9. Memorization and Forgetting
  10. Experiments
  11. Diverging and converging condition.
  12. Relation between memorization and forgetting.
  13. Impact Statement
  14. Appendix
  15. Additional related works
  16. ...and 10 more sections

Key Result

Lemma 3.5

Consider the unlearning update operator $J_k$ defined in eq:Jk-def. Define a sequence of PSD matrices $\{N_k\}_{k\ge0}$ by $N_0 = I$ and for $k\ge 1$: with $C_r, C_f$ as given in Definition def:mixHessian. Also let $M_k = J^{2k} + N_k$. ($J= I -\eta (1 - \alpha) H_R + \eta \alpha H_F$ where $H_R$ and $H_F$ ar full Hessian of retain and forget set. See definition def: full forget and retain hessia

Figures (2)

  • Figure 1: Tight upper and lower bounds. Blue = convergence, red = divergence. The dashed line is the lower bound (Theorem \ref{['thm:convergence']}), the solid line the divergence criterion (Theorem \ref{['thm:divergence']}). Both closely track the true boundary.
  • Figure 2: (Left) Training loss. (Middle) Test error. (Right) Forget loss. Memorization and forgetting regions strongly overlap as indicated by the blue region.

Theorems & Definitions (24)

  • Definition 3.1: Coherence, single set dexter2024precisecharacterizationsgdstability
  • Definition 3.2: Mix-Hessian
  • Definition 3.3: Mix-coherence matrix
  • Definition 3.4: Unlearning Coherence Measure
  • Lemma 3.5: Stability recurrence for unlearning
  • Theorem 3.6: Divergence criterion for unlearning
  • Theorem 3.7: Convergence condition (matching lower bound)
  • Definition 3.8: Data Setup
  • Theorem 3.9: Coherence bound in the CNN memorization model
  • Definition A.1: Full forget Hessian and retain Hessian
  • ...and 14 more