CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learning

TL;DR

CAPrompt tackles prompt inconsistency in class incremental learning by removing the need for task-ID prediction through cyclic prompt aggregation. It proves that aggregating prompts under a concavity condition can reduce prediction error and enforces this condition with a concave and a linear constraint, while further boosting performance with a cyclic prompt-weight strategy that iteratively refines weights. The method yields 2–3% improvements over state-of-the-art on CIFAR-100, ImageNet-R, and CUB-200 and demonstrates strong robustness across backbones and datasets. Overall, CAPrompt offers a task-ID agnostic, knowledge-sharing framework that leverages learned prompts more effectively to mitigate forgetting in continual vision tasks.

Abstract

Recently, prompt tuning methods for pre-trained models have demonstrated promising performance in Class Incremental Learning (CIL). These methods typically involve learning task-specific prompts and predicting the task ID to select the appropriate prompts for inference. However, inaccurate task ID predictions can cause severe inconsistencies between the prompts used during training and inference, leading to knowledge forgetting and performance degradation. Additionally, existing prompt tuning methods rely solely on the pre-trained model to predict task IDs, without fully leveraging the knowledge embedded in the learned prompt parameters, resulting in inferior prediction performance. To address these issues, we propose a novel Cyclic Prompt Aggregation (CAPrompt) method that eliminates the dependency on task ID prediction by cyclically aggregating the knowledge from different prompts. Specifically, rather than predicting task IDs, we introduce an innovative prompt aggregation strategy during both training and inference to overcome prompt inconsistency by utilizing a weighted sum of different prompts. Thorough theoretical analysis demonstrates that under concave conditions, the aggregated prompt achieves lower error compared to selecting a single task-specific prompt. Consequently, we incorporate a concave constraint and a linear constraint to guide prompt learning, ensuring compliance with the concave condition requirement. Furthermore, to fully exploit the prompts and achieve more accurate prompt weights, we develop a cyclic weight prediction strategy. This strategy begins with equal weights for each task and automatically adjusts them to more appropriate values in a cyclical manner. Experiments on various datasets demonstrate that our proposed CAPrompt outperforms state-of-the-art methods by 2%-3%. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-CAPrompt.

Figures (8)

• Figure 1: Most existing prompt-based methods (Dualprompt) predict task ID during inference which may cause inconsistency between the prompts during training and inference. In contrast, we propose a prompt aggregation strategy to eliminate the requirement to predict task ID. Moreover, a cyclic prompt weight strategy is proposed to adjust the weights of different prompts.
• Figure 2: The overall pipeline of our proposed CAPrompt. To overcome the inconsistency of prompts between the training phase (a) and inference phase (b), a prompt aggregation strategy is proposed. The concave and linear constraints are proposed to guarantee the prediction error of the aggregated prompt is the lower bound of using one task-specific prompt. Then, to make full use of prompts in predicting weight for prompt aggregation, we propose a cyclic prompt weight that we initiate the prompt with equal weight and cyclically predict the prompt weight. This strategy can be conducted cyclically many times to further improve performance.
• Figure 3: Motivation of Linear Constraint.
• Figure 4: The complete classification accuracy of different methods on each task.
• Figure 5: Accuracy of the proposed method increases with the number of cycles ($num$).