Differential Privacy in Continual Learning: Which Labels to Update?

TL;DR

This work exposes a privacy vulnerability in differential-privacy–enabled continual learning: releasing the classifier's output space can leak information about sensitive data. It formalizes task-wise DP for continual learning and proposes two data-independent strategies for the output label space—Sprior and Slearned—to preserve DP guarantees while maintaining utility. The authors instantiate DP-CL with pre-trained-model approaches, developing a Cosine Similarity Classifier and a PEFT Ensemble, and demonstrate robust performance across varied priors, blurry tasks, and domain shifts. The results offer practical guidance for privacy-preserving CL, enabling effective use of large label taxonomies and pre-trained representations under DP constraints.

Abstract

The goal of continual learning (CL) is to retain knowledge across tasks, but this conflicts with strict privacy required for sensitive training data that prevents storing or memorising individual samples. To address that, we combine CL and differential privacy (DP). We highlight that failing to account for privacy leakage through the set of labels a model can output can break the privacy of otherwise valid DP algorithms. This is especially relevant in CL. We show that mitigating the issue with a data-independent overly large label space can have minimal negative impact on utility when fine-tuning a pre-trained model under DP, while learning the labels with a separate DP mechanism risks losing small classes.

Figures (15)

• Figure 1: Attack on the output label space: The challenger uses either the dataset $\mathcal{D}_t$ or the adjacent $\mathcal{D}_t'$ (one more point ) for training a classifier $f_t$ or $f_t'$. The output label space of the classifier is released to the attacker and the attacker can guess the dataset. Observing the classifier output space can leak catastrophically when one of the datasets contains one more label .
• Figure 2: Lower bound of the probability that a new label is not added to the output label space ${\mathcal{O}}_t$. Even with $\epsilon=1.0$ and $\delta=10^{-7}$, classes having fewer than $13$ samples are discarded with at least 99% probability, and thus cannot be learned.
• Figure 3: Split-CIFAR-100 at $\epsilon$=$1$ (left) and Split-ImageNet-R at $\epsilon$=$8$ (right) at $\delta$=$10^{-5}$: Both methods only decrease slightly in utility when greatly increasing the number of assumed labels through dummy labels. Bad label match affects Cosine Classifier. Similar observations and detailed results in \ref{['app:add_results_label']}.
• Figure 4: Blurry tasks on Split-CIFAR-100 (left: data distribution per task, right: final acc with $\delta=10^{-5}$): Cosine Classifier is unaffected by the task blurriness (constant accuracy) as it is invariant but the PEFT Ensemble is especially effected by Si-Blurry (Tabular results in \ref{['tab:blurry']}).
• Figure 5: Split-CIFAR-100 (left) and Split-ImageNet-R (right) with $\delta$=$10^{-5}$: PEFT Ensemble outperforms, but Cosine Classifier feasible at lower domain shift (left). Detailed results in \ref{['app:cifar100_additional', 'app:imagenet-r', 'app:5-dataset']}

Theorems & Definitions (27)

• Definition 3.1: DP; dwork2006calibratingdwork2006epsilondelta
• Proposition 4.1
• Proposition 4.2
• Definition 5.1: Task-wise DP
• Lemma 5.2
• Definition B.1: Definition 2 in desai2021continual
• Definition B.2: Definition 3 in Lai2022LifelongDP
• Proposition C.1
• proof
• proof