Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracy

TL;DR

This work reveals a critical vulnerability in machine unlearning by showing that adversarial forget sets not drawn from the training data can catastrophically degrade a model after unlearning. It introduces a threat model with white-box and black-box attack methods that compute adversarial forget sets via gradient-through-unlearning and zeroth-order estimation, and validates them on CIFAR-10 and ImageNet against GA-family unlearning methods. The results show dramatic accuracy losses (white-box: CIFAR-10 ~3.6%, ImageNet ~0.4%; black-box: CIFAR-10 ~8.5%, ImageNet ~1.3%) and reveal transferability across models, highlighting significant robustness gaps. Defensive analyses demonstrate that existing verification schemes struggle to reliably detect adversarial requests, especially under stealthy perturbations, underscoring an urgent need for stronger request verification and secure unlearning protocols for practical deployment.

Abstract

Machine unlearning algorithms, designed for selective removal of training data from models, have emerged as a promising approach to growing privacy concerns. In this work, we expose a critical yet underexplored vulnerability in the deployment of unlearning systems: the assumption that the data requested for removal is always part of the original training set. We present a threat model where an attacker can degrade model accuracy by submitting adversarial unlearning requests for data not present in the training set. We propose white-box and black-box attack algorithms and evaluate them through a case study on image classification tasks using the CIFAR-10 and ImageNet datasets, targeting a family of widely used unlearning methods. Our results show extremely poor test accuracy following the attack: 3.6% on CIFAR-10 and 0.4% on ImageNet for white-box attacks, and 8.5% on CIFAR-10 and 1.3% on ImageNet for black-box attacks. Additionally, we evaluate various verification mechanisms to detect the legitimacy of unlearning requests and reveal the challenges in verification, as most of the mechanisms fail to detect stealthy attacks without severely impairing their ability to process valid requests. These findings underscore the urgent need for research on more robust request verification methods and unlearning protocols, should the deployment of machine unlearning systems become more prevalent in the future.

Figures (7)

• Figure 1: Machine unlearning allows data owners to remove their training data from a target model without compromising the unlearned model’s accuracy on examples not subject to unlearning requests, such as test data (left). However, we demonstrate that adversarially crafted unlearning requests, though visually similar to legitimate ones, can lead to a catastrophic drop in model accuracy after unlearning (right).
• Figure 2: Visualization of the original forget set $\mathcal{D}_\textrm{forget}$ (a, c) and adversarial forget sets $\mathcal{D}_\textrm{forget}^\text{adv}$ (b, d) for CIFAR-10 and ImageNet. Although the adversarial forget sets appear visually similar to the original valid forget set, they lead to catastrophic accuracy failure in the unlearned model.
• Figure 3: Accuracy drop ($\%$) across different targeted classes in CIFAR-10 after targeted attack, with impact on non-targeted classes kept $< 10\%$.
• Figure 4: ROC curves for various detection methods under (a) non-stealthy attacks and (b) stealthy attacks, where adversarial changes are projected onto a valid input space constrained by an $\ell_2$ ball. The stealthiness of adversarial forget sets makes them harder to differentiate from benign requests with natural perturbations, resulting in lower AUC.
• Figure 5: The attacker can optimize the selection of examples in a valid forget set to degrade performance. We report the maximum, mean, and minimum retain error on CIFAR-10 for varying $|\mathcal{D}_\textrm{forget}|$, with 3000 random selections per size.

Theorems & Definitions (2)

• Theorem C.1: Adversarial Forget Sets
• proof : Proof of \ref{['thm:adv-forget-theory']}