Subtract the Corruption: Training-Data-Free Corrective Machine Unlearning using Task Arithmetic

TL;DR

This paper tackles post-training corruption by formalizing source-free corrective machine unlearning (CMU), where the original training data are unavailable. It introduces CUTS, a lightweight method that treats corruption as a distinct task and estimates a corruption vector in weight space using a small proxy dataset; the correction is performed by subtracting a calibrated multiple of this vector from the mixed-model weights. The approach is demonstrated across label-noise and backdoor-candidate corruptions, various architectures, and real-world data, showing substantial recovery of utility and robust backdoor removal with minimal data. The work highlights the role of pre-trained priors for disentangling clean and corruption directions and suggests practical avenues for zero-shot proxy generation and layer-wise edits in future work.

Abstract

Corrupted training data are ubiquitous. Corrective Machine Unlearning (CMU) seeks to remove the influence of such corruption post-training. Prior CMU typically assumes access to identified corrupted training samples (a "forget set"). However, in many real-world scenarios the training data are no longer accessible. We formalize source-free CMU, where the original training data are unavailable and, consequently, no forget set of identified corrupted training samples can be specified. Instead, we assume a small proxy (surrogate) set of corrupted samples that reflect the suspected corruption type without needing to be the original training samples. In this stricter setting, methods relying on forget set are ineffective or narrow in scope. We introduce Corrective Unlearning in Task Space (CUTS), a lightweight weight space correction method guided by the proxy set using task arithmetic principles. CUTS treats the clean and the corruption signal as distinct tasks. Specifically, we briefly fine-tune the corrupted model on the proxy to amplify the corruption mechanism in the weight space, compute the difference between the corrupted and fine-tuned weights as a proxy task vector, and subtract a calibrated multiple of this vector to cancel the corruption. Without access to clean data or a forget set, CUTS recovers a large fraction of the lost utility under label noise and, for backdoor triggers, nearly eliminates the attack with minimal damage to utility, outperforming state-of-the-art specialized CMU methods in source-free setting.

Figures (12)

• Figure 1: The training distribution mixes clean and corrupted distributions. Classic CMU corrects via an identified forget set of corrupted training samples. In source-free setting, the training dataset is inaccessible; one can only use a proxy set of samples from the corruption distribution not necessarily seen during training.
• Figure 2: Task vector subtraction from a pre-trained model induces partial task forgetting, as demonstrated for partial unlearning of toxic language and degrading OCR performance in ilharco2023editing. Our motivation is the following: if corruption can be modeled as a task, task arithmetic enables source-free, proxy-guided CMU.
• Figure 3: High-level view of CUTS correction procedure in the weight space.
• Figure 4: RR% of CUTS on ResNet models with different initializations on CIFAR10 with symmetric label noise at $\eta{=}40\%$.
• Figure 5: PCA of the penultimate features of CLIP for $\theta_\text{mix}- \alpha {\tau}_p$ , varying $\alpha \in \{ 0, \hat{\alpha}^*/3, 2 \hat{\alpha}^*/3, \hat{\alpha}^*\}$. Top: CIFAR10 under symmetric label noise at $\eta{=}60\%$. Bottom: CIFAR10 under poison trigger corruption at $\eta{=}2\%$ with target class airplane.