Certifiable Machine Unlearning for Linear Models

TL;DR

This work addresses certifiable machine unlearning for linear models trained with SGD by experimentally comparing three state-of-the-art approximate methods—Fisher-based, Influence-based, and DeltaGrad—across six real-world datasets. It presents a unified ML pipeline with training, inference, unlearning, auditing, and an online retraining strategy to decide when full retraining is warranted. The study reveals trade-offs among efficiency, effectiveness, and certifiability, showing Fisher and Influence often achieve strong certifiability with substantial speed-ups, while DeltaGrad offers stability with slower performance in high-dimensional settings. The proposed online retraining criterion and systematic evaluation provide practical guidance for deploying certifiable unlearning in real systems and motivate extending the analysis to data addition and more complex models.

Abstract

Machine unlearning is the task of updating machine learning (ML) models after a subset of the training data they were trained on is deleted. Methods for the task are desired to combine effectiveness and efficiency, i.e., they should effectively "unlearn" deleted data, but in a way that does not require excessive computation effort (e.g., a full retraining) for a small amount of deletions. Such a combination is typically achieved by tolerating some amount of approximation in the unlearning. In addition, laws and regulations in the spirit of "the right to be forgotten" have given rise to requirements for certifiability, i.e., the ability to demonstrate that the deleted data has indeed been unlearned by the ML model. In this paper, we present an experimental study of the three state-of-the-art approximate unlearning methods for linear models and demonstrate the trade-offs between efficiency, effectiveness and certifiability offered by each method. In implementing the study, we extend some of the existing works and describe a common ML pipeline to compare and evaluate the unlearning methods on six real-world datasets and a variety of settings. We provide insights into the effect of the quantity and distribution of the deleted data on ML models and the performance of each unlearning method in different settings. We also propose a practical online strategy to determine when the accumulated error from approximate unlearning is large enough to warrant a full retrain of the ML model.

Figures (26)

• Figure 1: The common ML pipeline with the three stages of training, inference and unlearning. First, an initial model $\mathbf{w}^{*}$ is trained on all data and used for inference; subsequently, whenever a part $\mathcal{D}_{m\xspace}$ of the data is deleted, an updated model $\mathbf{w}\xspace^u\xspace$ is obtained via machine unlearning. The pipeline restarts if the updated model is deemed inadequate.
• Figure 2: Effect of different deletion distributions on the test accuracy ($\text{\tt Acc}_{\text{test}}$) and deleted data accuracy ($\text{\tt Acc}_{\text{del}}$) as the fraction of deleted data is varied for different datasets.
• Figure 3: Efficiency-Certifiability trade-offs for (a) $\sigma\xspace=0$ at all volumes and (b) $\sigma\xspace=1$ at the largest volume of deletion as efficiency parameter is varied. The y-axis reports certifiability (AccDis) and the x-axis reports the efficiency (speed-up).
• Figure 4: Efficiency-Effectiveness trade-off at $\sigma\xspace=1$ for the largest volume of deletion as $\tau$ is varied. The y-axis reports effectiveness (AccErr) and the x-axis reports efficiency (speed-up). The legend is same as in \ref{['fig:certifiability-efficiency']}.
• Figure 5: Effectiveness-Certifiability trade-offs for largest volume of deletion. Each row corresponds to an unlearning method. Efficiency parameter is fixed at $m\xspace^{\prime}=m$ for Influence and Fisher and $T_0=100$ for DeltaGrad. Left y-axis reports certifiability (AccDis), right y-axis reports effectiveness (AccErr) and x-axis varies noise parameter $\sigma$. Lower is better for both y-axes.