Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearning

TL;DR

This work addresses the practical utility gap in certified unlearning by introducing Sequential Subspace Noise Injection (SSNI), a block-wise extension of noisy fine-tuning that distributes the privacy-preserving noise across orthogonal parameter subspaces. By conditioning on a proximity bound $\Delta(\rho)$ between the fully trained and retrained models, the authors obtain tighter, more practical guarantees while maintaining the $(\varepsilon,\delta)$ certification; they also prove that the same privacy budget is preserved under the block decomposition. The method yields improved post-unlearning accuracy across MNIST, CIFAR-10, and ViT-Tiny settings, with strong robustness to membership inference attacks and competitive or superior unlearning metrics compared to baselines. Overall, SSNI demonstrates that certified unlearning can achieve rigorous guarantees without sacrificing practical utility, enabling scalable and reliable forgetting in deep networks.

Abstract

Certified unlearning based on differential privacy offers strong guarantees but remains largely impractical: the noisy fine-tuning approaches proposed so far achieve these guarantees but severely reduce model accuracy. We propose sequential noise scheduling, which distributes the noise budget across orthogonal subspaces of the parameter space, rather than injecting it all at once. This simple modification mitigates the destructive effect of noise while preserving the original certification guarantees. We extend the analysis of noisy fine-tuning to the subspace setting, proving that the same $(\varepsilon,δ)$ privacy budget is retained. Empirical results on image classification benchmarks show that our approach substantially improves accuracy after unlearning while remaining robust to membership inference attacks. These results show that certified unlearning can achieve both rigorous guarantees and practical utility.

Figures (8)

• Figure 1: Severe accuracy drop under noisy fine-tuning. On CIFAR-10 with ResNet-18, standard noisy fine-tuning (NFT, koloskova2025certified) test accuracy drops sharply from $98\%$ to below $20\%$ once the unlearning begins, and does not recover even after 1000 subsequent fine-tuning steps.
• Figure 2: Illustration of the intuition behind the negative result for Noisy Fine-Tuning. The illustration shows fully-trained model $\textbf{x}_0$, retrained model $\textbf{x}'_0$ and $\textbf{x}_{\rm init}$ after model clipping. Unlearning trajectories require $T < T_{\rm retrain}$ steps. However, we need at least $T_{\rm retrain}$ steps of unlearning on the $\textbf{x}_{\rm init}$ to reach good quality (the green region). Therefore, results of unlearning ($T$ steps from $\textbf{x}_{\rm init}$) cannot obtain a good quality.
• Figure 3: Random 10% deletion on MNIST and CIFAR-10. We compare standard Noisy Fine-Tuning (NFT) with Block-wise NFT (k=2,4,10) with the final retrain accuracy shown for reference. Across privacy budgets Block-wise NFT shows smoother, more stable unlearning and better post–fine-tuning recovery; increasing $k$ further reduces early accuracy loss.
• Figure 4: Block-construction schemes for Block-wise NFT on MNIST at $\varepsilon\in\{0.5,1.0\}$ and $\delta=10^{-5}$.
• Figure 5: Block-construction schemes for Block-wise NFT on CIFAR--10 at $\varepsilon=5.0$ and $\delta=10^{-5}$.

Theorems & Definitions (25)

• Definition 1: ($\varepsilon, \delta$)-unlearning koloskova2025certified
• Definition 2: Noisy fine-tuning, koloskova2025certified
• Theorem 1: Per-step noise lower bound
• Proposition 1
• Remark 1
• Definition 3: High-Probability Initial Discrepancy
• Remark 2
• Remark 3
• Proposition 2
• Remark 4