Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Unlearning with Model Updates Probably Aligned with Gradients

Virgile Dine, Teddy Furon, Charly Faure

TL;DR

This work reframes machine unlearning as a constrained optimization problem that balances forgetting protected data with preserving utility. It introduces feasible updates based on masking and batch-gradient noise modeling, forming add-ons that can be plugged into existing first-order unlearning methods. The authors provide a theoretical foundation via KKT conditions and a probabilistic masking framework, including a focus vector to guide updates under gradient uncertainty. Extensive experiments across 360 configurations on CV tasks demonstrate improved privacy metrics such as MIA and relative unlearning accuracy with manageable trade-offs in accuracy and runtime. The proposed approach offers a practical, principled way to enhance unlearning methods without retraining from scratch, with broad applicability to privacy, robustness, and fairness concerns in real-world systems.

Abstract

We formulate the machine unlearning problem as a general constrained optimization problem. It unifies the first-order methods from the approximate machine unlearning literature. This paper then introduces the concept of feasible updates as the model's parameter update directions that help with unlearning while not degrading the utility of the initial model. Our design of feasible updates is based on masking, \ie\ a careful selection of the model's parameters worth updating. It also takes into account the estimation noise of the gradients when processing each batch of data to offer a statistical guarantee to derive locally feasible updates. The technique can be plugged in, as an add-on, to any first-order approximate unlearning methods. Experiments with computer vision classifiers validate this approach.

Improving Unlearning with Model Updates Probably Aligned with Gradients

TL;DR

This work reframes machine unlearning as a constrained optimization problem that balances forgetting protected data with preserving utility. It introduces feasible updates based on masking and batch-gradient noise modeling, forming add-ons that can be plugged into existing first-order unlearning methods. The authors provide a theoretical foundation via KKT conditions and a probabilistic masking framework, including a focus vector to guide updates under gradient uncertainty. Extensive experiments across 360 configurations on CV tasks demonstrate improved privacy metrics such as MIA and relative unlearning accuracy with manageable trade-offs in accuracy and runtime. The proposed approach offers a practical, principled way to enhance unlearning methods without retraining from scratch, with broad applicability to privacy, robustness, and fairness concerns in real-world systems.

Abstract

We formulate the machine unlearning problem as a general constrained optimization problem. It unifies the first-order methods from the approximate machine unlearning literature. This paper then introduces the concept of feasible updates as the model's parameter update directions that help with unlearning while not degrading the utility of the initial model. Our design of feasible updates is based on masking, \ie\ a careful selection of the model's parameters worth updating. It also takes into account the estimation noise of the gradients when processing each batch of data to offer a statistical guarantee to derive locally feasible updates. The technique can be plugged in, as an add-on, to any first-order approximate unlearning methods. Experiments with computer vision classifiers validate this approach.

Paper Structure

This paper contains 32 sections, 7 theorems, 37 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Machine Unlearning
  4. Membership Inference Attack
  5. Gradient Agreements
  6. Problem Statement
  7. Notations
  8. Exact and Approximate Unlearning
  9. Usual Evaluation Metrics
  10. General Framework for First-Order Methods
  11. Optimization Problem
  12. KKT Condition at Equilibrium
  13. Feasible Update
  14. Processing with Batches
  15. Probabilistic Modeling of Gradients
  16. ...and 17 more sections

Key Result

Proposition 1

In the optimization problem problem, the KKT conditions impose the necessary condition at the equilibrium $\bar{\theta}$:

Figures (9)

  • Figure 1: Masking procedure to ensure agreement between losses with $\mathrm{Agg}$\ref{['eq:MinG']} in pink and \ref{['eq:linG']} in blue ($\alpha=0.8$, $\beta=0.5$).
  • Figure 2: Masking wrong modification direction caused by noisy gradient, with $\mathrm{Agg}$\ref{['eq:linG']} for the updates ($\alpha=0.8$, $\beta=0.5$).
  • Figure 3: Kernel density estimation and frequencies over values. Distribution of the agreeing probabilities $\mathbb{P}\left[g_{U,i}g_{C,i} > 0 \,|\, \hat{g}_{U}, \hat{g}_{C}\right]$ for all the parameters of a VGG16 during an unlearning epoch step. Doted vertical line at $p = 1/2$.
  • Figure 4: Heatmap of the probability $\mathbb{P}\left[g_{U,i}g_{C,i}>0 \,|\, \hat{g}_{U}, \hat{g}_{C}\right]$ with respect to coordinates $(\phi_{U,i}, \phi_{C,i})\in(0,1)^2$. Hatched areas correspond to a probability lower than $p$, provoking a masking by \ref{['prob-selection']} - $p=0.45$ in this example.
  • Figure 5: Comparison of add-ons (SalUn, $\mathrm{AND}$, $\mathrm{PROB}$ and $\mathrm{F}$) over SRL method. Evolution of the metrics over the number of epochs. Cifar10 -- VGG16 -- 50% Random forgetting
  • ...and 4 more figures

Theorems & Definitions (15)

  • Proposition 1: Necessary Condition at Equilibrium
  • Definition 1
  • Proposition 2: Feasible update's Guarantee
  • Proposition 3: Vicinity of $\theta_0$
  • Proposition 4: Gradient's Probable Alignment
  • Proposition 5: Properties of $m_{\mathrm{PROB}}$
  • Proposition 6: Focus update's Theoretical Guarantee
  • Proposition 7: Properties of the Focus Vector $f$
  • proof
  • proof
  • ...and 5 more