ACU: Analytic Continual Unlearning for Efficient and Exact Forgetting with Privacy Preservation

TL;DR

This work tackles Continual Unlearning (CU) within continual learning (CL) by identifying limitations of gradient-based unlearning that require access to retained data. It introduces Analytic Continual Unlearning (ACU), a gradient-free framework that uses a frozen pre-trained backbone and a linear analytic classifier trained via a least-squares objective to derive closed-form updates for each unlearning request. ACU maintains a Knowledge Tracking Matrix $\mathbf{T}$ and updates the model $\mathbf{W}$ in a way that is theoretically equivalent to retraining on the retained data, without ever accessing it, ensuring exact forgetting and data privacy. Empirical results on CIFAR-10/100 show ACU outperforms gradient-based baselines in unlearning effectiveness, model fidelity, and system efficiency, especially under frequent CU requests, while requiring orders of magnitude less computation than re-training from scratch.

Abstract

The development of artificial intelligence demands that models incrementally update knowledge by Continual Learning (CL) to adapt to open-world environments. To meet privacy and security requirements, Continual Unlearning (CU) emerges as an important problem, aiming to sequentially forget particular knowledge acquired during the CL phase. However, existing unlearning methods primarily focus on single-shot joint forgetting and face significant limitations when applied to CU. First, most existing methods require access to the retained dataset for re-training or fine-tuning, violating the inherent constraint in CL that historical data cannot be revisited. Second, these methods often suffer from a poor trade-off between system efficiency and model fidelity, making them vulnerable to being overwhelmed or degraded by adversaries through deliberately frequent requests. In this paper, we identify that the limitations of existing unlearning methods stem fundamentally from their reliance on gradient-based updates. To bridge the research gap at its root, we propose a novel gradient-free method for CU, named Analytic Continual Unlearning (ACU), for efficient and exact forgetting with historical data privacy preservation. In response to each unlearning request, our ACU recursively derives an analytical (i.e., closed-form) solution in an interpretable manner using the least squares method. Theoretical and experimental evaluations validate the superiority of our ACU on unlearning effectiveness, model fidelity, and system efficiency.

Figures (3)

• Figure 1: Illustration of model state evolution under the CL and CU phases. The model begins blank with no knowledge. As the CL phase progresses, the model incrementally acquires knowledge from online tasks. After the CL phase, the CU requests are sequentially issued to remove particular knowledge acquired during the CL phase. Since the online data from the CL phase is discarded after training, only the designated samples to be forgotten can be accessible for each CU request.
• Figure 2: The framework of our proposed ACU method.
• Figure 3: The dynamics analysis for a total of 25 CU requests.