A Certified Unlearning Approach without Access to Source Data

TL;DR

The paper tackles certifiably removing data from trained models without access to the original training data by introducing a surrogate-data framework that mimics source statistics. It develops a second-order Newton unlearning update using Hessians estimated from the surrogate, coupled with a Gaussian mechanism whose noise scales with the statistical distance between source and surrogate distributions to achieve (ε,δ)-certified unlearning. A theoretical bound ties the forecasted model drift to the distance measure, and the methodology is paired with practical estimation strategies (KL-based proxies, energy-based sampling, and DV bounds) to operate without direct source data. Extensive experiments on synthetic and real-world datasets demonstrate effective unlearning with utility comparable to data-access baselines while providing rigorous privacy guarantees.

Abstract

With the growing adoption of data privacy regulations, the ability to erase private or copyrighted information from trained models has become a crucial requirement. Traditional unlearning methods often assume access to the complete training dataset, which is unrealistic in scenarios where the source data is no longer available. To address this challenge, we propose a certified unlearning framework that enables effective data removal \final{without access to the original training data samples}. Our approach utilizes a surrogate dataset that approximates the statistical properties of the source data, allowing for controlled noise scaling based on the statistical distance between the two. \updated{While our theoretical guarantees assume knowledge of the exact statistical distance, practical implementations typically approximate this distance, resulting in potentially weaker but still meaningful privacy guarantees.} This ensures strong guarantees on the model's behavior post-unlearning while maintaining its overall utility. We establish theoretical bounds, introduce practical noise calibration techniques, and validate our method through extensive experiments on both synthetic and real-world datasets. The results demonstrate the effectiveness and reliability of our approach in privacy-sensitive settings.

Figures (4)

• Figure 1: (a): Required variance $\sigma$ for achieving certified unlearning on synthetic datasets as a function of the off-diagonal elements ($\zeta$). (b): Forget scores achieved for synthetic datasets.
• Figure 2: (a): Required variance $\sigma$ for achieving certified unlearning across CIFAR10, StanfordDogs, and Caltech256 datasets as a function of the concentration parameter $\xi$. (b): Forget scores achieved for CIFAR10, StanfordDogs, and Caltech256.
• Figure 3: (a) Required noise variance $\sigma$ for certified unlearning on synthetic data as a function of the off-diagonal elements of the covariance matrix ($\zeta$). Both exact and heuristic (approximate) estimates are shown based on KL divergence. (b) Estimated KL divergence vs. $\zeta$. Exact values use the closed-form Gaussian KL divergence, approximate values use our heuristic based on model parameters and surrogate data. (c) Forget scores achieved for synthetic datasets for varying off-diagonal elements of the covariance matrix ($\zeta$).
• Figure 4: (a) Required noise variance $\sigma$ for certified unlearning on StanfordDogs as a function of the Dirichlet parameter ($\xi$). Both exact and heuristic (approximate) estimates are shown based on KL divergence. (b) Estimated KL divergence vs. $\xi$. Exact values are calculated by using the exact and surrogate data samples, approximate values use our heuristic based on model parameters and surrogate data. (c) Forget scores achieved for StanfordDogs dataset for varying Dirichlet parameter ($\xi$).

Theorems & Definitions (33)

• Definition 3.1: $(\epsilon, \delta)$-Certified Unlearning sekhari_remember_2021
• Theorem 4.2
• proof
• Theorem 4.3
• proof
• Corollary 4.4
• proof
• Proposition 4.5
• proof
• Proposition 4.6