Semantic Residual Prompts for Continual Learning

TL;DR

STAR-Prompt tackles catastrophic forgetting in prompt-based continual learning by combining CLIP-guided stability with a two-level prompting framework that couples a stable first level to a plastic second level. Class-specific keys $\mathbf{w}_c = E_{txt}([p_c; \operatorname{[CLS-NAME]}])$ guide a second-level prompt pool $Q_c$ to generate a semantic residual $\mathbf{R}$ that is added to each ViT layer via $\mathbf{e}' = \operatorname{MSA}(\operatorname{LN}(\mathbf{e}_l)) + \mathbf{e}_l + \mathbf{R}[l]$, enabling plastic adaptation while keeping the CLIP space stable. The method also introduces MoG-based generative replay across CLIP and ViT features with losses ${\cal L}_{GR}^P$ and ${\cal L}_{GR}^Q$, and orthogonality constraints to minimize cross-task interference. Experiments across nine diverse datasets, including satellite and medical domains with substantial domain shifts, show STAR-Prompt outperforms state-of-the-art CLIP-based prompting approaches and approaches the performance of joint, stationary training, highlighting strong stability-plasticity balance and practical applicability.

Abstract

Prompt-tuning methods for Continual Learning (CL) freeze a large pre-trained model and train a few parameter vectors termed prompts. Most of these methods organize these vectors in a pool of key-value pairs and use the input image as query to retrieve the prompts (values). However, as keys are learned while tasks progress, the prompting selection strategy is itself subject to catastrophic forgetting, an issue often overlooked by existing approaches. For instance, prompts introduced to accommodate new tasks might end up interfering with previously learned prompts. To make the selection strategy more stable, we leverage a foundation model (CLIP) to select our prompts within a two-level adaptation mechanism. Specifically, the first level leverages a standard textual prompt pool for the CLIP textual encoder, leading to stable class prototypes. The second level, instead, uses these prototypes along with the query image as keys to index a second pool. The retrieved prompts serve to adapt a pre-trained ViT, granting plasticity. In doing so, we also propose a novel residual mechanism to transfer CLIP semantics to the ViT layers. Through extensive analysis on established CL benchmarks, we show that our method significantly outperforms both state-of-the-art CL approaches and the zero-shot CLIP test. Notably, our findings hold true even for datasets with a substantial domain gap w.r.t. the pre-training knowledge of the backbone model, as showcased by experiments on satellite imagery and medical datasets. The codebase is available at https://github.com/aimagelab/mammoth.

Figures (6)

• Figure 1: Comparison between current approaches (left) and our STAR-Prompt (right), regarding their prompting selection strategies. To enhance the stability of the pool selection, we exploit the multi-modal CLIP embedding space and compute a similarity between the image and prompt-learned class-prototype keys. Afterwards, the retrieved values are used as prompts for another backbone, i.e. an ImageNet pre-trained ViT.
• Figure 2: The architecture of STAR-Prompt. The bottom left box illustrates our CL setting, in which first- and second-level prompts of the old tasks are frozen.
• Figure 3:
• Figure 4:
• Figure A: Analysis of prompt selection stability for Split Cars-196, Split EuroSAT, Split CUB-200, and Split CropDiseases. We assess various models at the end of the last task and report results as confusion matrices. The $y$ axis indicates the task of the query sample, while the $x$ axis shows the task of the corresponding selected key.