Forgetting Similar Samples: Can Machine Unlearning Do it Better?
Heng Xu, Tianqing Zhu, Dayong Ye, Lefeng Zhang, Le Wang, Wanlei Zhou
TL;DR
The paper investigates machine unlearning when training data contains target samples and their similar counterparts, arguing that many methods forget only the target sample rather than its full influence. It formalizes influence via mutual information and introduces similarity-entailed datasets to stress-test forgetting, revealing that existing schemes—and even retraining-from-scratch baselines—often leave residual influence from similar samples. To address this, the authors propose robustness-training–inspired enhancements for both image and language unlearning, including expanded unlearning sets and manifold smoothing with KL regularization, which improve forgetting while preserving performance. The study provides practical insights and a public implementation to spur more robust unlearning techniques in privacy-sensitive settings.
Abstract
Machine unlearning, a process enabling pre-trained models to remove the influence of specific training samples, has attracted significant attention in recent years. Although extensive research has focused on developing efficient machine unlearning strategies, we argue that these methods mainly aim at removing samples rather than removing samples' influence on the model, thus overlooking the fundamental definition of machine unlearning. In this paper, we first conduct a comprehensive study to evaluate the effectiveness of existing unlearning schemes when the training dataset includes many samples similar to those targeted for unlearning. Specifically, we evaluate: Do existing unlearning methods truly adhere to the original definition of machine unlearning and effectively eliminate all influence of target samples when similar samples are present in the training dataset? Our extensive experiments, conducted on four carefully constructed datasets with thorough analysis, reveal a notable gap between the expected and actual performance of most existing unlearning methods for image and language models, even for the retraining-from-scratch baseline. Additionally, we also explore potential solutions to enhance current unlearning approaches.