FaLW: A Forgetting-aware Loss Reweighting for Long-tailed Unlearning

TL;DR

This work addresses machine unlearning under long-tailed forgetting distributions, revealing two failure modes: heterogeneous and skewed unlearning deviation. It introduces FaLW, a forgetting-aware, per-sample dynamic loss reweighting that compares a sample’s current prediction to an estimated unseen-data distribution and adjusts forgetting pressure with a balancing factor that emphasizes tail classes. The method, which is plug-and-play and compatible with existing unlearning approaches, demonstrates state-of-the-art performance across CIFAR-100, CIFAR-10, and Tiny-ImageNet under various forget ratios and imbalance levels, with extensive ablations and visualization supporting its effectiveness. By enabling more faithful and privacy-preserving unlearning in realistic long-tailed settings, FaLW has practical impact for complying with data-rights regulations in large-scale models.

Abstract

Machine unlearning, which aims to efficiently remove the influence of specific data from trained models, is crucial for upholding data privacy regulations like the ``right to be forgotten". However, existing research predominantly evaluates unlearning methods on relatively balanced forget sets. This overlooks a common real-world scenario where data to be forgotten, such as a user's activity records, follows a long-tailed distribution. Our work is the first to investigate this critical research gap. We find that in such long-tailed settings, existing methods suffer from two key issues: \textit{Heterogeneous Unlearning Deviation} and \textit{Skewed Unlearning Deviation}. To address these challenges, we propose FaLW, a plug-and-play, instance-wise dynamic loss reweighting method. FaLW innovatively assesses the unlearning state of each sample by comparing its predictive probability to the distribution of unseen data from the same class. Based on this, it uses a forgetting-aware reweighting scheme, modulated by a balancing factor, to adaptively adjust the unlearning intensity for each sample. Extensive experiments demonstrate that FaLW achieves superior performance. Code is available at \textbf{Supplementary Material}.

Figures (7)

• Figure 1: (a) Distributions of a 20% CIFAR-100 forget set generated via balanced, long-tailed, and random sampling (Seeds 1-3 denote different runs). (b) Predicted probability distributions for SalUn vs. Retrain (ResNet-18) after unlearning a 20% randomly sampled forget set from CIFAR-100.
• Figure 2: Under the setup of the ResNet-18 architecture trained on CIFAR-100 dataset with a 30% unlearning ratio and a forget set distribution defined by $\mathcal{N}_k \propto \frac{1}{k^{\gamma}}$, this figure visualizes forgetting accuracy gaps between Salun, our method (Ours), and the Retrain baseline across head, mid, and tail classes within the forget set.
• Figure 3: The FaLW framework. Left: Analyzing instance-level unlearning bias using the model's predictive probability on unseen data. Right: The top shows instance-level loss reweighting, and the bottom explains the forgetting-aware weight and balance factor.
• Figure 4: Comparative results of predicted probability distributions on randomly sampled forgotten data between the Salun and Retrain methods under multiple settings.
• Figure 5: Performance discrepancies between the SalUn method and Retrain on the head, middle, and tail segments of forgotten data under different long-tailed configurations across various experimental settings.

Theorems & Definitions (7)

• Definition 1: Unlearning Deviation
• Proposition 1
• Proposition 2
• Proposition 1: Proposed in paper
• proof : Proof
• Proposition 2: Proposed in paper
• proof