A Unified Framework for Continual Learning and Unlearning

TL;DR

This work tackles the need for models that can learn new tasks while selectively forgetting others. It introduces UniCLUN, a unified framework based on a multi-teacher, single-student architecture with a bounded replay buffer and controlled knowledge distillation, combining CL and UL through a context-sensitive objective ${\mathcal{L}} = \gamma {\mathcal{L}}_{cl} + (1-\gamma) {\mathcal{L}}_{cu}$. The CL module employs a bounded replay buffer and losses ${\mathcal{L}}_{ce}$, ${\mathcal{L}}_{od}$, ${\mathcal{L}}_{cd}$, ${\mathcal{L}}_{scd}$, while the UL module uses a bad teacher with a KL-divergence-based objective to erase unwanted knowledge. Empirical results on CIFAR-10 and ciFAIR-10 across various CL-UL distributions show strong retention of retained tasks and effective unlearning, outperforming standard CL baselines on preserved classes and matching or surpassing UL baselines in forgetting focused settings. The framework thus enables adaptable models capable of dynamic learning and forgetting with strong overall performance, signaling a practical path for robust, privacy-aware, and regulatory-compliant continual systems.

Abstract

Continual learning and machine unlearning are crucial challenges in machine learning, typically addressed separately. Continual learning focuses on adapting to new knowledge while preserving past information, whereas unlearning involves selectively forgetting specific subsets of data. In this paper, we introduce a new framework that jointly tackles both tasks by leveraging controlled knowledge distillation. Our approach enables efficient learning with minimal forgetting and effective targeted unlearning. By incorporating a fixed memory buffer, the system supports learning new concepts while retaining prior knowledge. The distillation process is carefully managed to ensure a balance between acquiring new information and forgetting specific data as needed. Experimental results on benchmark datasets show that our method matches or exceeds the performance of existing approaches in both continual learning and machine unlearning. This unified framework is the first to address both challenges simultaneously, paving the way for adaptable models capable of dynamic learning and forgetting while maintaining strong overall performance. Source code: \textcolor{blue}{https://respailab.github.io/CLMUL}

Figures (2)

• Figure 1: Comparing the isolated problems of continual learning and unlearning with the unified problem of learning and unlearning in the same framework as proposed in this paper.
• Figure 2: The proposed controlled knowledge distillation based unified framework for continual learning and unlearning.

Theorems & Definitions (1)

• Definition 1