MCU: Improving Machine Unlearning through Mode Connectivity

TL;DR

This work addresses the challenge of unlearning specific training data without compromising overall model utility by moving beyond linear parameter updates. It introduces Mode Connectivity Unlearning (MCU), a nonlinear MU framework that searches a Bézier-path in parameter space between the original model and a pre-unlearning model, yielding a spectrum of unlearning models along the path. MCU incorporates a two-part parameter mask to restrict updates to critical parameters and an adaptive unlearning penalty that balances forgetting quality with predictive performance without manual hyperparameter tuning. Empirical results on image classification demonstrate that MCU consistently improves unlearning efficacy across datasets and architectures while maintaining competitive running times, and its adaptive variant closely matches retraining baselines in class-wise forgetting scenarios, indicating strong practical impact for privacy-preserving MU.

Abstract

Machine Unlearning (MU) aims to remove the information of specific training data from a trained model, ensuring compliance with privacy regulations and user requests. While one line of existing MU methods relies on linear parameter updates via task arithmetic, they suffer from weight entanglement. In this work, we propose a novel MU framework called Mode Connectivity Unlearning (MCU) that leverages mode connectivity to find an unlearning pathway in a nonlinear manner. To further enhance performance and efficiency, we introduce a parameter mask strategy that not only improves unlearning effectiveness but also reduces computational overhead. Moreover, we propose an adaptive adjustment strategy for our unlearning penalty coefficient to adaptively balance forgetting quality and predictive performance during training, eliminating the need for empirical hyperparameter tuning. Unlike traditional MU methods that identify only a single unlearning model, MCU uncovers a spectrum of unlearning models along the pathway. Overall, MCU serves as a plug-and-play framework that seamlessly integrates with any existing MU methods, consistently improving unlearning efficacy. Extensive experiments on the image classification task demonstrate that MCU achieves superior performance.

Figures (9)

• Figure 1: Overview of our proposed MCU framework. (a) Identify a parameter mask by first filtering out the top $k_r$ proportion of parameters important to the retaining data, then reserving the top $k$ proportion of parameters crucial to the forgetting data. (b) Explore nonlinear pathways in the parameter space, where $\theta_c$ serves as the control point shaping the pathway. (c) Locate the optimal unlearning model and an effective unlearning region along the pathway.
• Figure 2: The accuracy on $\mathcal{D}_f$, $\mathcal{D}_r$, and $\mathcal{D}_t$ across different retaining data proportions used in our training process. It shows that all accuracy performance remains stable even with scarce retaining data.
• Figure 3: The efficiency and effectiveness of our parameter mask. 'w/o' and 'w/' in the left panel represent the results without $10\%$ mask and with $10\%$ mask. The x-axis of the right panel represents the parameter $t$ along the Bézier curve, while the y-axis corresponds to accuracy.
• Figure 4: Effective unlearning region on $\text{MCU}_{\beta}$. The marker ★ highlights the position with the minimum average gap from RT, with the accompanying numerical value indicating the exact average accuracy gap of $\mathcal{D}_f$, $\mathcal{D}_r$ and $\mathcal{D}_t$ (and $\mathcal{D}_{tf}$ for class-wise forgetting). The dotted line represents the RT method's accuracy, serving as a reference. The shaded gray area denotes the effective unlearning region, where models achieve better unlearning performance than the pre-unlearning model.
• Figure 5: Ablation study for fixed $\beta$ on MCU. Overall, increasing $\beta$ effectively enhances the unlearning effect but damages retaining predictive performance, while decreasing $\beta$ weakens the ability of the pathway to forget data.