SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness

TL;DR

This work tackles label-noise degradation by introducing Scaled Activation Projection (SAP), a corrective unlearning method that performs a single, computation-efficient weight update. SAP identifies a small set of trusted samples via cross-entropy loss, constructs activation representations from them, and uses SVD to derive a clean activation space that is encoded into a Projection Matrix $W_p$; the final update is $\widehat{W} = W W_p^T$. By scaling the SVD components with an importance matrix $\Lambda$ determined by a hyperparameter $\alpha$, SAP aligns network activations away from corrupted signals without explicit data forgetting, achieving consistent improvements on CIFAR-10/100 under multiple noise types and offering gains when combined with robust learning methods and real-world noisy datasets. The approach is validated across CNNs and a Vision Transformer, demonstrating practical utility for scalable, data-efficient unlearning in noisy training environments.

Abstract

Label corruption, where training samples are mislabeled due to non-expert annotation or adversarial attacks, significantly degrades model performance. Acquiring large, perfectly labeled datasets is costly, and retraining models from scratch is computationally expensive. To address this, we introduce Scaled Activation Projection (SAP), a novel SVD (Singular Value Decomposition)-based corrective machine unlearning algorithm. SAP mitigates label noise by identifying a small subset of trusted samples using cross-entropy loss and projecting model weights onto a clean activation space estimated using SVD on these trusted samples. This process suppresses the noise introduced in activations due to the mislabeled samples. In our experiments, we demonstrate SAP's effectiveness on synthetic noise with different settings and real-world label noise. SAP applied to the CIFAR dataset with 25% synthetic corruption show upto 6% generalization improvements. Additionally, SAP can improve the generalization over noise robust training approaches on CIFAR dataset by ~3.2% on average. Further, we observe generalization improvements of 2.31% for a Vision Transformer model trained on naturally corrupted Clothing1M.

Figures (8)

• Figure 1: Sources of label noise - Intentional/Unintentional Errors.
• Figure 2: Decision boundaries for a network trained the 2D spiral data (a) with full Clean data, (b) with $10\%$ corrupt data and (c) SAP applied to network trained on corrupt data.
• Figure 3: Overview of the proposed SAP algorithm. Here, $\theta_*$ denotes the model weights which comprises of the layer weights represented by $W$.
• Figure 4: (a) Singular value distribution obtained through SVD on the Representative activations. (b) Effect of $\alpha$ in importance scaling Equation \ref{['eqn:scaling']} and (c) the projection of noisy activation on Activation Alignment Space to obtain clean activations.
• Figure 5: Effect of trusted sample size for CIFAR100 dataset with $\eta=25\%$.