PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer

TL;DR

This work tackles catastrophic forgetting in rehearsal-free, memory-constrained incremental learning by leveraging a single cross-task prompt mechanism. The proposed PECTP framework introduces two retention modules: PRM to constrain cross-task prompt evolution via Outer Prompt Granularity and Inner Prompt Granularity losses, and HRM to inherit old classifier knowledge, enabling the prompts to generalize across all incremental tasks. Empirical results on seven diverse datasets show that PECTP achieves higher final and cumulative accuracies while using far fewer learnable parameters and eliminating prompt-selection overhead, compared with prompt-extending and prompt-fixed baselines. The approach offers a practical, memory-efficient pathway for continual learning with large pre-trained vision models and has potential for hybridization with other prompt-based IL strategies in future work.

Abstract

Incremental Learning (IL) aims to learn deep models on sequential tasks continually, where each new task includes a batch of new classes and deep models have no access to task-ID information at the inference time. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples (rehearsal-free) and with a memory constraint (memory-constrained): Prompt-extending and Prompt-fixed methods. However, prompt-extending methods need a large memory buffer to maintain an ever-expanding prompt pool and meet an extra challenging prompt selection problem. Prompt-fixed methods only learn a single set of prompts on one of the incremental tasks and can not handle all the incremental tasks effectively. To achieve a good balance between the memory cost and the performance on all the tasks, we propose a Parameter-Efficient Cross-Task Prompt (PECTP) framework with Prompt Retention Module (PRM) and classifier Head Retention Module (HRM). To make the final learned prompts effective on all incremental tasks, PRM constrains the evolution of cross-task prompts' parameters from Outer Prompt Granularity and Inner Prompt Granularity. Besides, we employ HRM to inherit old knowledge in the previously learned classifier heads to facilitate the cross-task prompts' generalization ability. Extensive experiments show the effectiveness of our method. The source codes will be available at \url{https://github.com/RAIAN08/PECTP}.

Figures (7)

• Figure 1: Comparison of different state-of-the-art incremental learning methods using Pre-Trained Models. The X-axis represents the total number of learnable parameters, and the Y-axis represents the average accuracy.
• Figure 2: Prompt-extending methods face concerns from two aspects. As incremental tasks keep increasing, the learnable parameters gradually increase (overhead problem). Simultaneously, as inference sessions progress, prompt selection accuracy becomes lower (selection problem).
• Figure 3: Illustration of sequential tasks with two incremental tasks. Prompt-fixed methods train a set of prompts $\mathcal{P}$ only on incremental task $1$, fix the parameters, and directly infer on the remaining incremental tasks. Prompt-extending methods learn a task-specific set of prompts $\mathcal{P}_{i}$ for each task. PECTP uses a single set of prompts but updates these learnable tokens on each incremental task, effectively retaining knowledge from previous tasks through two meticulously designed modules, PRM and HRM, which act on the single set of prompts and the classifier head, respectively.
• Figure 4: An architecture of the PECTP framework. When learning task $k$, a training sample is processed by both $\Phi_{k-1}$ and $\Phi_{k}$ to extract image features $\mathbf{h}^{k-1}$ and $\mathbf{h}^{k}$, respectively. In this context, $\Phi_{k-1}$ uses a fixed set of prompts $\mathcal{P}^{k-1}$, while $\Phi_{k}$ employs a learnable set of prompts $\mathcal{P}^{k}$. PRM applies constraints from the Outer Prompt Granularity on $\phi_{n,w,h}$, $\phi_{w,h}$, $\phi_{n,h}$ and $\phi_{n,w}$ (OPG). Additionally, PRM imposes constraints directly on the parameters of the single set of prompts (IPG). Moreover, HRM transfers knowledge from previously learned task-specific classifier heads to initialize the classifier head for task $k$.
• Figure 5: The detailed improvement of PECTP over the baseline for each task. The x-axis denotes each incremental task. Apart from ADAM-VPT-Deep, we also present the results of PlainCIL and SimpleCIL.