$\nabla τ$: Gradient-based and Task-Agnostic machine Unlearning

TL;DR

Gradient-based and Task-Agnostic machine Unlearning ($\nabla \tau$) presents an adaptive gradient framework for removing the influence of a forget set while preserving model performance. By combining an adaptive ReLU-based memory loss term with standard retain-set optimization, the method becomes model- and task-agnostic and robust to forget-set size, demonstrating forgetting up to 30% of data across image and text domains without hyperparameter tuning. Evaluation via Membership Inference Attacks shows up to a 10% improvement over state-of-the-art baselines while maintaining accuracy, supporting practical privacy guarantees under GDPR-like constraints. The work formalizes a weak unlearning objective, analyzes robustness to hyperparameters, and demonstrates effectiveness in random subset removal and class-removal scenarios, paving the way for scalable, privacy-preserving unlearning in real-world systems.

Abstract

Machine Unlearning, the process of selectively eliminating the influence of certain data examples used during a model's training, has gained significant attention as a means for practitioners to comply with recent data protection regulations. However, existing unlearning methods face critical drawbacks, including their prohibitively high cost, often associated with a large number of hyperparameters, and the limitation of forgetting only relatively small data portions. This often makes retraining the model from scratch a quicker and more effective solution. In this study, we introduce Gradient-based and Task-Agnostic machine Unlearning ($\nabla τ$), an optimization framework designed to remove the influence of a subset of training data efficiently. It applies adaptive gradient ascent to the data to be forgotten while using standard gradient descent for the remaining data. $\nabla τ$ offers multiple benefits over existing approaches. It enables the unlearning of large sections of the training dataset (up to 30%). It is versatile, supporting various unlearning tasks (such as subset forgetting or class removal) and applicable across different domains (images, text, etc.). Importantly, $\nabla τ$ requires no hyperparameter adjustments, making it a more appealing option than retraining the model from scratch. We evaluate our framework's effectiveness using a set of well-established Membership Inference Attack metrics, demonstrating up to 10% enhancements in performance compared to state-of-the-art methods without compromising the original model's accuracy.

Figures (2)

• Figure 1: Loss distributions of Forget (in orange) and Test (in blue) sets before (left) and after (right) unlearning procedure on CIFAR-10.
• Figure 2: Experiment using our method on CIFAR-10 across different forget set sizes (Y Axis) and $\alpha$ parameter (X Axis). On the left we report the absolute distance of $MIA_{L}$ from the ideal value 50%. To improve the readability of the heatmaps, we do not report the standard deviations. However, for results where $|MIA_L - 50|< 1.0$ (the ones that best approximate retraining), the standard deviation is always under 1%. Note that even the golden baseline has a standard deviation $\pm0.5\%$ from 50%. On the right, we report the absolute difference between the accuracy on forget set $A_{D_f}$ and the accuracy on Test set $A_{D_t}$. The results highlight a similar pattern, indicating that similar scores in the accuracies are correlated to lower MIA scores. The results show the mean across three runs with different seeds.