On Conformal Machine Unlearning

TL;DR

A new definition for MU is introduced based on conformal prediction (CP), providing statistically sound, uncertainty-aware guarantees without the need for the concept of naive retraining, and a practical unlearning method designed to optimize these conformal metrics.

Abstract

The increasing demand for data privacy has made machine unlearning (MU) essential for removing the influence of specific training samples from machine learning models while preserving performance on retained data. However, most existing MU methods lack rigorous statistical guarantees or rely on heuristic metrics such as accuracy. To overcome these limitations, we introduce a new definition for MU based on conformal prediction (CP), providing statistically sound, uncertainty-aware guarantees without the need for the concept of naive retraining. We formalize the proposed conformal criteria that quantify how often forgotten samples are excluded from CP sets, and propose empirical metrics to measure the effectiveness of unlearning. We further present a practical unlearning method designed to optimize these conformal metrics. Extensive experiments across diverse forgetting scenarios, datasets and models demonstrate the efficacy of our approach in removing targeted data.

Figures (11)

• Figure 1: Plot of the train ($\calT_f$) accuracy and validation ($\calV_f$) accuracy and coverage over the forgotten data vs. the number of forgotten clusters in CIFAR100 for the retrained ($A^r$) and original ($A^o$) models.
• Figure 1: Accuracy on forgotten and retained data for 100 samples random forgetting in AG-News. $A^r$ is accuracy from retrained model and $A^o$ from original model.
• Figure 2: Conformal workflow.
• Figure 3: CIFAR100, targeted 10-class forgetting with $\alpha=0.05$ and $c=d=100$ (CQMU). The shifts are applied after normalization. Left: metrics vs mean shift with std scaling fixed to 2. Right: metrics vs std scaling with mean shift fixed to 2.
• Figure 4: Imagenet100 targeted class-wise forgetting with $c,d=40$, $\alpha=0.05$, and 10 forgotten class.

Theorems & Definitions (5)

• proof
• proof
• proof
• proof
• proof