Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, Xiaolin Huang

TL;DR

This work proposes a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction, free from specific modal constraints, and adaptable across computer vision unlearning tasks, including classification and generation.

Abstract

Machine unlearning (MU) has emerged to enhance the privacy and trustworthiness of deep neural networks. Approximate MU is a practical method for large-scale models. Our investigation into approximate MU starts with identifying the steepest descent direction, minimizing the output Kullback-Leibler divergence to exact MU inside a parameters' neighborhood. This probed direction decomposes into three components: weighted forgetting gradient ascent, fine-tuning retaining gradient descent, and a weight saliency matrix. Such decomposition derived from Euclidean metric encompasses most existing gradient-based MU methods. Nevertheless, adhering to Euclidean space may result in sub-optimal iterative trajectories due to the overlooked geometric structure of the output probability space. We suggest embedding the unlearning update into a manifold rendered by the remaining geometry, incorporating second-order Hessian from the remaining data. It helps prevent effective unlearning from interfering with the retained performance. However, computing the second-order Hessian for large-scale models is intractable. To efficiently leverage the benefits of Hessian modulation, we propose a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction. Free from specific modal constraints, our approach is adaptable across computer vision unlearning tasks, including classification and generation. Extensive experiments validate our efficacy and efficiency. Notably, our method successfully performs class-forgetting on ImageNet using DiT and forgets a class on CIFAR-10 using DDPM in just 50 steps, compared to thousands of steps required by previous methods.

Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

TL;DR

This work proposes a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction, free from specific modal constraints, and adaptable across computer vision unlearning tasks, including classification and generation.

Abstract

Machine unlearning (MU) has emerged to enhance the privacy and trustworthiness of deep neural networks. Approximate MU is a practical method for large-scale models. Our investigation into approximate MU starts with identifying the steepest descent direction, minimizing the output Kullback-Leibler divergence to exact MU inside a parameters' neighborhood. This probed direction decomposes into three components: weighted forgetting gradient ascent, fine-tuning retaining gradient descent, and a weight saliency matrix. Such decomposition derived from Euclidean metric encompasses most existing gradient-based MU methods. Nevertheless, adhering to Euclidean space may result in sub-optimal iterative trajectories due to the overlooked geometric structure of the output probability space. We suggest embedding the unlearning update into a manifold rendered by the remaining geometry, incorporating second-order Hessian from the remaining data. It helps prevent effective unlearning from interfering with the retained performance. However, computing the second-order Hessian for large-scale models is intractable. To efficiently leverage the benefits of Hessian modulation, we propose a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction. Free from specific modal constraints, our approach is adaptable across computer vision unlearning tasks, including classification and generation. Extensive experiments validate our efficacy and efficiency. Notably, our method successfully performs class-forgetting on ImageNet using DiT and forgets a class on CIFAR-10 using DDPM in just 50 steps, compared to thousands of steps required by previous methods.
Paper Structure (33 sections, 6 theorems, 51 equations, 11 figures, 12 tables, 1 algorithm)

This paper contains 33 sections, 6 theorems, 51 equations, 11 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Approximate MU from Perspective of Steepest Descent
  4. Proposed Method
  5. Experiments
  6. Conclusion
  7. Detailed Proof
  8. Proof of Equation \ref{['eq:steepest descent rewrite']}
  9. Proof of Proposition \ref{['prop:steepest euclidean']}
  10. Proof of Proposition \ref{['prop:steepest KL']}
  11. Proof of Proposition \ref{['prop:implicit hessian approx']}
  12. Related Works
  13. Related Works on Machine Unlearning
  14. Related Works on Steepest Descent in Optimization
  15. Preliminary on Diffusion Model
  16. ...and 18 more sections

Key Result

Proposition 1

Under the Euclidean manifold metric, $\rho(\theta_t,\theta_{t+1})=\frac{1}{2}\lVert \theta_t-\theta_{t+1} \rVert^2$. Assuming that the current model $\theta_t=\mathop{\arg\min}_{\theta}\mathcal{L}^f(\theta;\boldsymbol{\varepsilon}_t)+\mathcal{L}^r(\theta)$. Let $H_*^f=\nabla^2\mathcal{L}^f(\theta_*;

Figures (11)

  • Figure 1: Overview of our proposal vs. previous unlearning methods on erasing concept 'nudity' in diffusion models gandikota2023erasingschramowski2023safe. Conventional methods seek the steepest descent within an Euclidean ball, often compromising general capabilities. In contrast, we reach the region around retraining along a remain-preserving manifold. To address the large cost of Hessian, we implicitly approximate the up-to-date salient unlearning direction.
  • Figure 2: Image generations for class-wise forgetting tasks on CIFAR-10 using DDPM by baselines and our proposed SFR-on along with ablation variants. The forgetting class is 'cat', 'I' refers to the generated image sample from this class, and 'C' denotes the remaining class name. More results can be found in Appendix \ref{['subsec:addexp gen']}.
  • Figure 3: Class-wise forgetting of 'golden retriever' in image generations of ImageNet with DiT, comparing baselines and our proposed SFR-on. RT$^\dag$ feeds autoencoder with random latent embeddings for the forgetting class, due to the computational constraints, rather than full RT. 'I' denotes image samples from forgetting, and 'C' refers to other remaining class name, e.g. 'Cacatua galerita' (C1). More results can be found in Appendix \ref{['subsec:addexp gen']}.
  • Figure A1: Performance of SFR-on with different $\lambda$ in adaptive coefficients $vs$ RT on CIFAR-10 using ResNet-18. The settings and metrics follow Tab.\ref{['tab:cls_cifar10_tinyimg_random_10']}. The points closer to RT and with lower $D_{\text{KL}}$ are better.
  • Figure A2: Performance of SFR-on with different $\gamma$ in weight saliency mask $vs$ RT on CIFAR-10 using ResNet-18. The settings and metrics follow Tab.\ref{['tab:cls_cifar10_tinyimg_random_10']}. The points closer to RT and with lower $D_{\text{KL}}$ are better.
  • ...and 6 more figures

Theorems & Definitions (10)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • proof
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof