Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Label Smoothing Improves Machine Unlearning

Zonglin Di, Zhaowei Zhu, Jinghan Jia, Jiancheng Liu, Zafar Takhirov, Bo Jiang, Yuanshun Yao, Sijia Liu, Yang Liu

TL;DR

MU aims to remove the influence of forgetting data with lower cost than retraining. The authors introduce UGradSL, a plug‑and‑play gradient-based unlearning method that couples gradient ascent on the forgetting data with gradient descent on retained data, augmented by generalized label smoothing (GLS) with a negative smoothing rate $\alpha<0$. They provide theoretical results showing when GA helps MU and how GLS enhances MU, including privacy considerations in the form of $\epsilon$-Label-LDP. Empirically, across six datasets and multiple forgetting paradigms, UGradSL achieves robust unlearning gains with modest additional computation, e.g., up to a $66\%$ improvement in unlearning accuracy over GA baselines, demonstrating a scalable approach to MU that balances forgetting strength and data utility.

Abstract

The objective of machine unlearning (MU) is to eliminate previously learned data from a model. However, it is challenging to strike a balance between computation cost and performance when using existing MU techniques. Taking inspiration from the influence of label smoothing on model confidence and differential privacy, we propose a simple gradient-based MU approach that uses an inverse process of label smoothing. This work introduces UGradSL, a simple, plug-and-play MU approach that uses smoothed labels. We provide theoretical analyses demonstrating why properly introducing label smoothing improves MU performance. We conducted extensive experiments on six datasets of various sizes and different modalities, demonstrating the effectiveness and robustness of our proposed method. The consistent improvement in MU performance is only at a marginal cost of additional computations. For instance, UGradSL improves over the gradient ascent MU baseline by 66% unlearning accuracy without sacrificing unlearning efficiency.

Label Smoothing Improves Machine Unlearning

TL;DR

MU aims to remove the influence of forgetting data with lower cost than retraining. The authors introduce UGradSL, a plug‑and‑play gradient-based unlearning method that couples gradient ascent on the forgetting data with gradient descent on retained data, augmented by generalized label smoothing (GLS) with a negative smoothing rate . They provide theoretical results showing when GA helps MU and how GLS enhances MU, including privacy considerations in the form of -Label-LDP. Empirically, across six datasets and multiple forgetting paradigms, UGradSL achieves robust unlearning gains with modest additional computation, e.g., up to a improvement in unlearning accuracy over GA baselines, demonstrating a scalable approach to MU that balances forgetting strength and data utility.

Abstract

The objective of machine unlearning (MU) is to eliminate previously learned data from a model. However, it is challenging to strike a balance between computation cost and performance when using existing MU techniques. Taking inspiration from the influence of label smoothing on model confidence and differential privacy, we propose a simple gradient-based MU approach that uses an inverse process of label smoothing. This work introduces UGradSL, a simple, plug-and-play MU approach that uses smoothed labels. We provide theoretical analyses demonstrating why properly introducing label smoothing improves MU performance. We conducted extensive experiments on six datasets of various sizes and different modalities, demonstrating the effectiveness and robustness of our proposed method. The consistent improvement in MU performance is only at a marginal cost of additional computations. For instance, UGradSL improves over the gradient ascent MU baseline by 66% unlearning accuracy without sacrificing unlearning efficiency.
Paper Structure (35 sections, 3 theorems, 34 equations, 6 figures, 15 tables, 1 algorithm)

This paper contains 35 sections, 3 theorems, 34 equations, 6 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Label Smoothing Enables Fast and Effective Unlearning
  4. Preliminary
  5. Gradient Ascent Can Help Gradient-Based Machine Unlearning
  6. Label Smoothing Improves MU
  7. Label smoothing helps and local differential privacy
  8. UGradSL: A Plug-and-Play and Gradient-Mixed MU Method
  9. Experiments and Results
  10. Experiment Setup
  11. Experiment Results
  12. Discussion
  13. Conclusions
  14. Roadmap
  15. Notation Table
  16. ...and 20 more sections

Key Result

Theorem 1

Given the approximation in Equation equ:param_distance, GA achieve exact MU if and only if where ${\bm H(\boldsymbol{\theta}_{r}^{*}, \boldsymbol{\theta}_{tr}^{*}) = \left(\sum_{z^{tr} \in D_{tr}} \nabla_{\boldsymbol{\theta}}^2 \ell(h_{\boldsymbol{\theta}_{r}^{*}}, z^{tr})\right) \left(\sum_{z^{r} \in D_r}\nabla^2_{\boldsymbol{\theta}} \ell(h_{\boldsymbol{\theta}_{tr}^{*}}, z^{r})\right)^

Figures (6)

  • Figure 1: The framework of UGradLS. When there is a unlearning request, we can split the $D_{tr}$ into $D_f$ and $D_r$. We first apply NLS on $z_i^f=\{x, y\} \in D_f$ to get $z_i^{\text{NLS}, \alpha}=\{x, y^{\text{NLS}, \alpha}\}$. In back-propagation process, we apply gradient descent on the data $z_{i}^r \in D_r$ and gradient ascent on the data smoothed $D_f$, which is the mix-gradient way.
  • Figure 2: Summary of the proposed method and baselines (CIFAR 100, random forgetting across all classes) in terms of the two different MU metrics and speed. The proposed method with a following (R) means the method is optimized according to the retrained model while the method without a (R) means the method is optimized according to the completely unlearningshah2023unlearning. For the plot of Average Gap, the lower left indicates better performance. Average Gap is to evaluate how close the approximate MU method is to the retrained model. The smaller the average gap is, the better MU performance is. The retrained model has no average gap as reference in this plot. The bold vertical dash shows the speed. When we target to make the approximate MU method close to the retrained model, UGradSL ($\color{violet}\filledstar$), UGradSL(R, $\color{gray}\filledstar$) and UGradSL+(R, $\color{yellow}\filledstar$) outperform the other methods. For the plot of Sum, the upper left indicates better performance. Sum is to evaluate the comprehensive performance of the approximate MU model. The higher the sum is, the better performance of the MU method is. The black dot represents the retrained model. When we target to improve the comprehensive performance of the approximate MU, UGradSL+($\color{red}\filledstar$) and UGradSL+(R, $\color{yellow}\filledstar$) outperform the other methods without too much delay. Moreover, in both plots, our proposed methods are always the top methods. Our methods show the robustness and generalization capability in both MU metrics.
  • Figure 3: The relationship between the performance and smooth rate in random forgetting across all classes using CIFAR-10. The gray dash line stands for the performance of retrain. Our methods can improve the unlearning accuracy (UA) without significant drop of testing accuracy (TA).
  • Figure 4: The confusion matrix of testing set and forgetting set $D_f$ using our method on CIFAR-10 with random forgetting across all the classes. There is no big difference between the prediction distribution. Our method will not make $D_f$ more distinguishable.
  • Figure 5: The distribution of $\langle \Delta \boldsymbol{\theta}_{r} - \Delta \boldsymbol{\theta}_{f}, \Delta \boldsymbol{\theta}_n-\Delta \boldsymbol{\theta}_{f}\rangle$ on CelebA dataset.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2
  • Definition 1: Label-LDP
  • Theorem 3
  • proof
  • proof
  • proof