Convolutional Prompting meets Language Models for Continual Learning

TL;DR

ConvPrompt introduces a convolutional prompt creation mechanism with layer-wise shared embeddings to enable both layer-specific learning and improved cross-task transfer in continual learning with vision transformers. It further leverages Large Language Models to generate fine-grained text descriptions of categories, enabling image- and attribute-based task similarity calculations that dynamically determine the number of prompts and promote knowledge sharing while controlling parameter overhead, via a regularization weight $\lambda$. Empirical results show a ew percent improvement over state-of-the-art with significantly fewer parameters, along with extensive ablations and sensitivity analyses (notably for $\lambda$) to disentangle module contributions. Overall, the approach offers a scalable, dynamic prompting framework for continual learning that balances performance and efficiency, with task similarity guiding prompt allocation and transfer across tasks.

Abstract

Continual Learning (CL) enables machine learning models to learn from continuously shifting new training data in absence of data from old tasks. Recently, pretrained vision transformers combined with prompt tuning have shown promise for overcoming catastrophic forgetting in CL. These approaches rely on a pool of learnable prompts which can be inefficient in sharing knowledge across tasks leading to inferior performance. In addition, the lack of fine-grained layer specific prompts does not allow these to fully express the strength of the prompts for CL. We address these limitations by proposing ConvPrompt, a novel convolutional prompt creation mechanism that maintains layer-wise shared embeddings, enabling both layer-specific learning and better concept transfer across tasks. The intelligent use of convolution enables us to maintain a low parameter overhead without compromising performance. We further leverage Large Language Models to generate fine-grained text descriptions of each category which are used to get task similarity and dynamically decide the number of prompts to be learned. Extensive experiments demonstrate the superiority of ConvPrompt and improves SOTA by ~3% with significantly less parameter overhead. We also perform strong ablation over various modules to disentangle the importance of different components.

Figures (2)

• Figure 1: Average Accuracy ($\mathbf{A_T}$) vs. $\mathbf{\lambda}$ value(Log Scaled) plot for ConvPrompt. The performance peaks at $1.00E-02$ and saturates thereafter.
• Figure 2: Inter-task attribute based task similarity calculation: For each attribute of the new task, the most similar attributes in the old tasks are found, and the corresponding cosine similarity values are computed (some of these have been shown above). The mean of all such max similarities of attributes in the new task, gives the overall task similarity.