A Robust Certified Machine Unlearning Method Under Distribution Shift

TL;DR

This work tackles certified machine unlearning under distribution shift caused by non-i.i.d. deletion requests. It identifies fundamental failures of traditional Newton-update unlearning when deletions bias the retained distribution and proposes TR-Certified Machine Unlearning: an iterative Newton-update framework constrained by a trust region that relies on local smoothness rather than global Lipschitz constants. The method yields closer approximations to retrained models and tighter residual-gradient bounds, enabling tighter $(\epsilon,\delta)$ guarantees with improved utility. Extensive experiments on MNIST and CIFAR-10 demonstrate superior utility-generalization and robust unlearning effectiveness under non-i.i.d. deletions, at a tractable computational cost relative to full retraining. This distributes a practical, distribution-aware route to certified unlearning in real-world, biased-deletion scenarios.

Abstract

The Newton method has been widely adopted to achieve certified unlearning. A critical assumption in existing approaches is that the data requested for unlearning are selected i.i.d.(independent and identically distributed). However,the problem of certified unlearning under non-i.i.d. deletions remains largely unexplored. In practice, unlearning requests are inherently biased, leading to non-i.i.d. deletions and causing distribution shifts between the original and retained datasets. In this paper, we show that certified unlearning with the Newton method becomes inefficient and ineffective under non-i.i.d. unlearning sets. We then propose a better certified unlearning approach by performing a distribution-aware certified unlearning framework based on iterative Newton updates constrained by a trust region. Our method provides a closer approximation to the retrained model and yields a tighter pre-run bound on the gradient residual, thereby ensuring efficient (epsilon, delta)-certified unlearning. To demonstrate its practical effectiveness under distribution shift, we also conduct extensive experiments across multiple evaluation metrics, providing a comprehensive assessment of our approach.

Figures (2)

• Figure 1: Relationship between KL divergence and $\Delta$F1 for CIFAR-10 with a base CNN model, following the method of zhang2024towards. Vertical error bars show variability across runs. The horizontal arrow highlights the transition from i.i.d. (left) to non-i.i.d. (right).
• Figure 2: Performance under distribution shift: (a) $\Delta$F1 comparison with zhang2024towards, (b) retrain vs. unlearned F1. Error bars show min--max ranges.

Theorems & Definitions (15)

• Definition 1
• Theorem 1
• Proposition 1: Distribution shift from biased deletion
• Proposition 2: Local Descent Lemma
• Definition 2: TR subproblem
• Definition 3
• Proposition 3: Pre-run gradient bound for TR
• Theorem 2
• Corollary 1
• proof : Proof of Proposition \ref{['prop:biased-deletion-simplified']}