Privacy-Aware Lifelong Learning

TL;DR

This work tackles the challenge of privacy-aware lifelong learning by unifying continual learning with exact unlearning in a single fixed-capacity model. The proposed Privacy-Aware Lifelong Learning (PALL) framework learns task-specific sparse subnetworks with parameter sharing, freezes past knowledge to prevent forgetting, and uses episodic memory rehearsal to enable exact unlearning with minimal memory overhead. PALL achieves zero forgetting and exact unlearning while preserving forward knowledge transfer, demonstrated across CNNs and vision transformers on image classification benchmarks, and shows favorable scalability and memory efficiency compared to independent models. These results advance responsible AI by enabling dynamic, privacy-compliant learning in real-world systems where data subjects may request deletion of their data.

Abstract

Lifelong learning algorithms enable models to incrementally acquire new knowledge without forgetting previously learned information. Contrarily, the field of machine unlearning focuses on explicitly forgetting certain previous knowledge from pretrained models when requested, in order to comply with data privacy regulations on the right-to-be-forgotten. Enabling efficient lifelong learning with the capability to selectively unlearn sensitive information from models presents a critical and largely unaddressed challenge with contradicting objectives. We address this problem from the perspective of simultaneously preventing catastrophic forgetting and allowing forward knowledge transfer during task-incremental learning, while ensuring exact task unlearning and minimizing memory requirements, based on a single neural network model to be adapted. Our proposed solution, privacy-aware lifelong learning (PALL), involves optimization of task-specific sparse subnetworks with parameter sharing within a single architecture. We additionally utilize an episodic memory rehearsal mechanism to facilitate exact unlearning without performance degradations. We empirically demonstrate the scalability of PALL across various architectures in image classification, and provide a state-of-the-art solution that uniquely integrates lifelong learning and privacy-aware unlearning mechanisms for responsible AI applications.

Figures (3)

• Figure 1: Illustration of PALL. Task-specific subnetworks obtained after learning are indicated by color. The subnetwork mask $\mathop{\mathrm{\mathbf{m}}}\nolimits_2$ for Task 2 contains two shared, frozen parameters from Task 1 (dashed blue lines), as well as $\mathbf{\overline{m}}_2$ (green connections). Following the unlearning request for Task 1, we reset all parameters trained on $\mathop{\mathrm{\mathcal{D}}}\nolimits^1$ (blue connections), and retrain any of those parameters which were used for knowledge transfer in later tasks (shown by red connections) using experience replay.
• Figure 2: Evaluations with different number of unlearning requests $N_u$ in $\mathop{\mathrm{\mathcal{R}}}\nolimits_{1:r}$ (averaged across 10 random seeds each), for the methods that perform retraining or finetuning after task unlearning. We excluded GEM since the performance was similar to ER. We use $\alpha=1/T$ for PALL.
• Figure A1: Comparisons of task learning durations on S-TinyImageNet ($T=100$), as the number of tasks increase. Methods indicated with solid lines (Indep. & PALL) can ensure exact unlearning upon request.