On Newton's Method to Unlearn Neural Networks

TL;DR

This work tackles the challenge of unlearning neural networks after data erasure by extending Newton's method to NN settings through cubic regularization. The authors identify severe Hessian degeneracy as the main obstacle and introduce CureNewton, a cubic-regularized surrogate that yields a robust, globally convergent update via a dual optimization over a damping parameter α. A stochastic variant, SCureNewton, reduces memory and compute demands, enabling practical unlearning on large models. Empirical results across batch and sequential unlearning tasks on diverse datasets and architectures show CureNewton and SCureNewton achieve competitive erasing quality and maintain post-unlearning performance, while offering theoretical justification and favorable running times relative to baselines.

Abstract

With the widespread applications of neural networks (NNs) trained on personal data, machine unlearning has become increasingly important for enabling individuals to exercise their personal data ownership, particularly the "right to be forgotten" from trained NNs. Since retraining is computationally expensive, we seek approximate unlearning algorithms for NNs that return identical models to the retrained oracle. While Newton's method has been successfully used to approximately unlearn linear models, we observe that adapting it for NN is challenging due to degenerate Hessians that make computing Newton's update impossible. Additionally, we show that when coupled with popular techniques to resolve the degeneracy, Newton's method often incurs offensively large norm updates and empirically degrades model performance post-unlearning. To address these challenges, we propose CureNewton's method, a principle approach that leverages cubic regularization to handle the Hessian degeneracy effectively. The added regularizer eliminates the need for manual finetuning and affords a natural interpretation within the unlearning context. Experiments across different models and datasets show that our method can achieve competitive unlearning performance to the state-of-the-art algorithm in practical unlearning settings, while being theoretically justified and efficient in running time.

Figures (7)

• Figure 1: The eigenspectrum (left) and the training rank dynamics (right) of $\mathbf{H}_{\mathbf{w}^*}^r$ in a 2-layer CNN trained on the FashionMNIST dataset (for $3$ random runs).
• Figure 2: Performance comparison in the sequential unlearning settings on different datasets and models. Top row: Llama-2 $\times$ AG-News. Bottom row: ResNet18 $\times$ CIFAR-10.
• Figure 3: $\alpha$ dynamics in CureNewton's method during sequential unlearning on FashionMNIST (for $3$ random runs).
• Figure 4: Loss distributions for samples on non-overfitted and overfitted, original and retraining models.
• Figure 5: Instance-level sequential unlearning of $10\%$ training data. Top row: Llama-2 $\times$ AG-News ($2000$ erased samples per round). Bottom row: ResNet-18 $\times$ CIFAR-10 ($1000$ erased samples per round).

Theorems & Definitions (4)

• Remark 4.1
• Remark 4.2
• Remark 5.1
• Remark 5.2