Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Roughness-Informed Machine Unlearning

Mohammad Partohaghighi, Roummel Marcia, Bruce J. West, YangQuan Chen

TL;DR

SRAGU addresses instability in approximate machine unlearning under large or adversarial forget deletions by reweighting AGU updates with layer-wise spectral stability signals derived from heavy-tailed weight spectra. The method maps per-layer tail exponents to bounded weights, concentrating updates in spectrally stable layers while damping brittle ones, yielding smoother unlearning trajectories and improved alignment to a gold retrained reference. Across MNIST, CIFAR-10/100, ImageNet-100, and UCI Adult with random, class-specific, and adversarial deletions, SRAGU achieves stronger forgetting proxies (lower $oldsymbol{ extepsilon_{ ext{pred}}}$ and $D_{ ext{KL}}$) and competitive retained accuracy, while maintaining modest computational overhead. This mechanism-driven approach advances scalable, privacy-aware unlearning suitable for regulated domains and lays groundwork for extensions to federated and large-scale models, with explicit diagnostics and ablations to validate the spectral weighting rationale.

Abstract

Machine unlearning aims to remove the influence of a designated forget set from a trained model while preserving utility on the retained data. In modern deep networks, approximate unlearning frequently fails under large or adversarial deletions due to pronounced layer-wise heterogeneity: some layers exhibit stable, well-regularized representations while others are brittle, undertrained, or overfit, so naive update allocation can trigger catastrophic forgetting or unstable dynamics. We propose Statistical-Roughness Adaptive Gradient Unlearning (SRAGU), a mechanism-first unlearning algorithm that reallocates unlearning updates using layer-wise statistical roughness operationalized via heavy-tailed spectral diagnostics of layer weight matrices. Starting from an Adaptive Gradient Unlearning (AGU) sensitivity signal computed on the forget set, SRAGU estimates a WeightWatcher-style heavy-tailed exponent for each layer, maps it to a bounded spectral stability weight, and uses this stability signal to spectrally reweight the AGU sensitivities before applying the same minibatch update form. This concentrates unlearning motion in spectrally stable layers while damping updates in unstable or overfit layers, improving stability under hard deletions. We evaluate unlearning via behavioral alignment to a gold retrained reference model trained from scratch on the retained data, using empirical prediction-divergence and KL-to-gold proxies on a forget-focused query set; we additionally report membership inference auditing as a complementary leakage signal, treating forget-set points as should-be-forgotten members during evaluation.

Statistical Roughness-Informed Machine Unlearning

TL;DR

SRAGU addresses instability in approximate machine unlearning under large or adversarial forget deletions by reweighting AGU updates with layer-wise spectral stability signals derived from heavy-tailed weight spectra. The method maps per-layer tail exponents to bounded weights, concentrating updates in spectrally stable layers while damping brittle ones, yielding smoother unlearning trajectories and improved alignment to a gold retrained reference. Across MNIST, CIFAR-10/100, ImageNet-100, and UCI Adult with random, class-specific, and adversarial deletions, SRAGU achieves stronger forgetting proxies (lower and ) and competitive retained accuracy, while maintaining modest computational overhead. This mechanism-driven approach advances scalable, privacy-aware unlearning suitable for regulated domains and lays groundwork for extensions to federated and large-scale models, with explicit diagnostics and ablations to validate the spectral weighting rationale.

Abstract

Machine unlearning aims to remove the influence of a designated forget set from a trained model while preserving utility on the retained data. In modern deep networks, approximate unlearning frequently fails under large or adversarial deletions due to pronounced layer-wise heterogeneity: some layers exhibit stable, well-regularized representations while others are brittle, undertrained, or overfit, so naive update allocation can trigger catastrophic forgetting or unstable dynamics. We propose Statistical-Roughness Adaptive Gradient Unlearning (SRAGU), a mechanism-first unlearning algorithm that reallocates unlearning updates using layer-wise statistical roughness operationalized via heavy-tailed spectral diagnostics of layer weight matrices. Starting from an Adaptive Gradient Unlearning (AGU) sensitivity signal computed on the forget set, SRAGU estimates a WeightWatcher-style heavy-tailed exponent for each layer, maps it to a bounded spectral stability weight, and uses this stability signal to spectrally reweight the AGU sensitivities before applying the same minibatch update form. This concentrates unlearning motion in spectrally stable layers while damping updates in unstable or overfit layers, improving stability under hard deletions. We evaluate unlearning via behavioral alignment to a gold retrained reference model trained from scratch on the retained data, using empirical prediction-divergence and KL-to-gold proxies on a forget-focused query set; we additionally report membership inference auditing as a complementary leakage signal, treating forget-set points as should-be-forgotten members during evaluation.
Paper Structure (92 sections, 27 equations, 2 figures, 24 tables, 2 algorithms)

This paper contains 92 sections, 27 equations, 2 figures, 24 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Context and Motivation
  3. Problem and Challenge
  4. Prior Work Positioning
  5. Key Gap: Layer-wise Heterogeneity and Optimization Geometry.
  6. Our Idea: SRAGU
  7. Contributions
  8. Roadmap
  9. Methodology
  10. Problem Formulation
  11. Baseline: Adaptive Gradient Unlearning (AGU)
  12. Sensitivity indicator.
  13. Normalization.
  14. Unlearning update.
  15. Stopping rule .
  16. ...and 77 more sections

Figures (2)

  • Figure 1: SRAGU overview. Statistical-Roughness Adaptive Gradient Unlearning (SRAGU) extends AGU by modulating parameter-wise sensitivities with a bounded layer-wise spectral stability weight.
  • Figure 2: Trajectory of $\epsilon_{\mathrm{pred}}(t)$. AGU exhibits a clear U-shaped degradation after reaching its minimum around step 60, while SRAGU continues to improve and remains more stable in later steps.