Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization

Aviv Shamsian, Eitan Shaar, Aviv Navon, Gal Chechik, Ethan Fetaya

TL;DR

OrthoGrad is proposed, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes, and demonstrates the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.

Abstract

Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the unlearn data with the gradient descent on a retain set representing the training data. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.

Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization

TL;DR

OrthoGrad is proposed, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes, and demonstrates the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.

Abstract

Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the unlearn data with the gradient descent on a retain set representing the training data. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.

Paper Structure

This paper contains 35 sections, 1 equation, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conflicting Gradients in Multi-Task Learning
  4. Background
  5. Method
  6. Geometric Motivation
  7. Practical Algorithm
  8. Experiments
  9. Baselines.
  10. Evaluation.
  11. Automatic Speech Recognition
  12. Ablation Study
  13. ASR Results
  14. Image Classification
  15. Robustness to retain size
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Illustration of the gradient orthogonalization process. The retain gradients $\mathbf{g}_r^1$, $\mathbf{g}_r^2$, and $\mathbf{g}_r^3$ (in blue) span a subspace (gray triangle). The projection vector $\mathbf{p}$ (in yellow) is obtained by applying QR decomposition on the retain gradients. The unlearn gradient $\mathbf{g}_u$ is projected using $\mathbf{p}$ to form unlearning gradient which is orthogonal to the retain subspace, $\mathbf{g}_u^{\perp}$.
  • Figure 2: Performance comparison of different gradient-based methods on ImageNet with ViT architecture. We report UIS values across varying numbers of retained samples.
  • Figure 3: Performance comparison of different gradient-based methods on ImageNet with ResNet18 architecture. We report UIS values across varying numbers of retained samples.

Theorems & Definitions (2)

  • Claim 1
  • Claim 2