Soft Weighted Machine Unlearning

TL;DR

The paper tackles over-unlearning in non-privacy machine unlearning by shifting from binary data removal to a soft-weighted, influence-based approach. It introduces a weighted influence function with per-sample weights $\epsilon_j$ and solves a convex quadratic program to discover optimal weights, enabling fine-grained model corrections that maintain utility while improving fairness and robustness. The framework is designed to be compatible with a wide range of unlearning methods and shows consistent improvements across multiple fairness and robustness tasks, including scenarios with large models, while adding negligible overhead. This work provides a practical and theoretically motivated path to mitigate over-unlearning and enhance the applicability of unlearning to real-world non-privacy objectives.

Abstract

Machine unlearning, as a post-hoc processing technique, has gained widespread adoption in addressing challenges like bias mitigation and robustness enhancement, colloquially, machine unlearning for fairness and robustness. However, existing non-privacy unlearning-based solutions persist in using binary data removal framework designed for privacy-driven motivation, leading to significant information loss, a phenomenon known as over-unlearning. While over-unlearning has been largely described in many studies as primarily causing utility degradation, we investigate its fundamental causes and provide deeper insights in this work through counterfactual leave-one-out analysis. In this paper, we introduce a weighted influence function that assigns tailored weights to each sample by solving a convex quadratic programming problem analytically. Building on this, we propose a soft-weighted framework enabling fine-grained model adjustments to address the over-unlearning challenge. We demonstrate that the proposed soft-weighted scheme is versatile and can be seamlessly integrated into most existing unlearning algorithms. Extensive experiments show that in fairness- and robustness-driven tasks, the soft-weighted scheme significantly outperforms hard-weighted schemes in fairness/robustness metrics and alleviates the decline in utility metric, thereby enhancing machine unlearning algorithm as an effective correction solution.

Figures (14)

• Figure 1: Actual Changes in Utility and Fairness/Robustness on Adult dataset for each sample's leave-one-out model. The X-axis represents the sample indices. The Y-axis for Fairness (Robustness) displays changes in demographic parity (adversarial loss) on the test set, with negative values indicating improved fairness (robustness) and positive values indicating reduced fairness (robustness). The Y-axis for Utility shows changes in test loss, with negative values indicating improved utility. Scatter points marked in Red indicate sample indices where fairness or robustness improves, but utility (generalization) declines.
• Figure 2: Illustration of difference of the proposed soft weighted vs. the hard weighted framework.
• Figure 3: Actual Changes vs. Approximate Changes. We evaluated the leave-one-out influence for all samples, with the First Row for LR and Second Row for the last layer of NN, on different performance metrics as follows: (Left) Model utility (loss on test set), (Middle) fairness (DP loss on test set), (Right) robustness (loss on adversarial sample).
• Figure 4: Hard Weighted Scheme vs. Soft Weighted Scheme. We use IF as the unlearning method to update model. The First Row for Fairness compares the hard- and soft-weighted schemes: A compares the weighting schemes with corresponding fairness influence values, B presents fairness and utility before and after applying hard-weighted IF, and C shows the same for soft-weighted IF. The Second Row for Robustness follows a similar structure: D compares the weighting schemes and corresponding robustness influence values, E presents robustness and utility before and after applying hard-weighted IF, and F shows the same for soft-weighted IF. Moreover, we use opacity to represent the value of weights.
• Figure 5: Performance on Fairness/Robustness Tasks. Different colors represent various unlearning algorithms: ● for the Hard-Weighted scheme and ✖ for the Soft-Weighted scheme. The First Two Rows (LR, NN) evaluate utility and fairness metrics, while The Last Two Rows (LR, NN) evaluate utility and robustness metrics across datasets. The Green Region highlights that Free Lunch cases occurs when unlearning improve both task performance and utility compared to original model. The soft weighting outperforms the hard weighting by enhancing task performance and mitigating decline in utility, even achieving free lunch in some of the unlearning algorithms.