Key-Value Pair-Free Continual Learner via Task-Specific Prompt-Prototype

TL;DR

This paper tackles continual learning without rehearsal by removing key-value prompt retrieval and introducing a task-specific prompt-prototype binding (ProP). ProP learns a task-specific prompt per task and derives a corresponding prototype that anchors the task's feature subspace, enabling direct, interference-resistant inference through similarity to prototypes. The approach is regularized at prompt initialization with an L2 term and trains using a cross-entropy loss, with prototypes ultimately serving as classifier weights. Across diverse datasets and backbones, ProP consistently outperforms existing prompt-based and rehearsal-free baselines, demonstrating strong anti-forgetting behavior and scalability for continual learning in real-world settings.

Abstract

Continual learning aims to enable models to acquire new knowledge while retaining previously learned information. Prompt-based methods have shown remarkable performance in this domain; however, they typically rely on key-value pairing, which can introduce inter-task interference and hinder scalability. To overcome these limitations, we propose a novel approach employing task-specific Prompt-Prototype (ProP), thereby eliminating the need for key-value pairs. In our method, task-specific prompts facilitate more effective feature learning for the current task, while corresponding prototypes capture the representative features of the input. During inference, predictions are generated by binding each task-specific prompt with its associated prototype. Additionally, we introduce regularization constraints during prompt initialization to penalize excessively large values, thereby enhancing stability. Experiments on several widely used datasets demonstrate the effectiveness of the proposed method. In contrast to mainstream prompt-based approaches, our framework removes the dependency on key-value pairs, offering a fresh perspective for future continual learning research.

Figures (5)

• Figure 1: Comparison of frameworks between mainstream prompt-based methods and ProP. Color green means trainable at training, while gray means frozen the components. (a) illustrates the inference process of mainstream prompt-based methods, which relies on key-value pairs to determine the prompt corresponding to the input. (b) shows the inference process of ProP, which binds task-specific prompt with prototype sharing same task id without the need for key-value pairs.
• Figure 2: Comparison of trainable parameters and average accuracy between ProP (Ours) and other CIL methods. “Using Exemplars” indicates storing samples from previous tasks for rehearsal. All experiments are conducted on ImageNet-R Init100 Inc50, using the same pre-trained model as the backbone.
• Figure 3: Last accuracy under different incremental settings, with the number of classes in the initial task kept the same as in the incremental tasks. All experiments are conducted using ViT-B/16-IN1K as the backbone.
• Figure 4: Results on different settings. Here, (a)(b) presents the average accuracy on large initial classes in ImageNet-R(a) and ImageNet-A(b) Init100 Inc50. All experiments are conducted based on ViT-B/16-IN21K. (c)(d) study the impact of components proposed in ProP, where "ViT" represents using the pre-trained ViT-B/16-IN21K for CIL without finetuning, while "Prompt-Prototype" indicates ViT-B/16-IN21K finetuned with task-specific prompt-prototype. "Lce" refers to training with only cross-entropy loss $\mathcal{L}_{CE}$ and "Lce+L2" denotes training with both cross-entropy loss $\mathcal{L}_{CE}$ and $L2$ loss $\mathcal{L}_{L2}$. (e)(f) show the influence of L2 loss coefficient and prompt length $L_p$ on model performance. Experiments are conducted on ImageNet-A Init20 Inc20 with ViT-B/16-IN21K as backbone.
• Figure 5: Visualization and average value. (a) and (b) are visualization of the feature and class prototypes at first and second stage on CIFAR Init5 Inc5. Test data is represented by dots while class prototypes are present by "✕", (c) is average value of last accuracy under different seeds based on ImageNet-R Init20 Inc20 with ViT-B/16-IN21K as backbone.