One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning

TL;DR

AdaPromptCL tackles continual learning under heterogeneous semantic shifts by introducing adaptive prompting through assign-and-refine semantic grouping. It maintains a variable number of semantic groups with group-wise prompts, reserves prospective groups, and refines groupings based on task semantics to balance knowledge transfer and forgetting. Empirical results show robust improvements across varying shifts and backbones, with notable gains over baselines and evidence that semantic refinement improves clustering quality and predictive accuracy. The approach offers practical benefits for real-world CL where task semantics evolve unpredictably, while also providing insights into computational trade-offs and prompt management strategies.

Abstract

In real-world continual learning (CL) scenarios, tasks often exhibit intricate and unpredictable semantic shifts, posing challenges for fixed prompt management strategies which are tailored to only handle semantic shifts of uniform degree (i.e., uniformly mild or uniformly abrupt). To address this limitation, we propose an adaptive prompting approach that effectively accommodates semantic shifts of varying degree where mild and abrupt shifts are mixed. AdaPromptCL employs the assign-and-refine semantic grouping mechanism that dynamically manages prompt groups in accordance with the semantic similarity between tasks, enhancing the quality of grouping through continuous refinement. Our experiment results demonstrate that AdaPromptCL outperforms existing prompting methods by up to 21.3%, especially in the benchmark datasets with diverse semantic shifts between tasks.

Figures (11)

• Figure 1: Key idea of AdaPromptCL: (a) shows the comparison of our adaptive prompting against existing prompting CL methods on a practical CL scenario; (b) depicts the inherent challenges in implementing adaptive prompting. Colors indicate task semantics.
• Figure 2: Performance of representative methods for different prompting strategies across various degrees of semantic shifts.
• Figure 3: Overall flow.
• Figure 4: Detailed view of the assign-and-refine semantic grouping process. I. The assignment step adds the task $\tau^t$ to the group $G_2^t$ and reserves a prospective group $\hat{G}^t_{1,1}$ with its prompt. II. The refinement step, assuming that a refinement is needed when the task $\tau^{t^\prime}$ is received, three groups $G^{t^\prime}_1$, $G^{t^\prime}_2$, and $G^{t^\prime}_3$ are reduced to fewer prospective groups $\hat{G}^t_{1,1}$ and $\hat{G}^t_{1,2}$, with their prompts retrieved.
• Figure 5: t-SNE visualization on the semantic groups formed by AdaPromptCL across 50 tasks on VTAB-Rec10. The tasks from the same semantic origin are in the same color, denoted by circles . The symbols × and × respectively represent the centroids of semantic groups with and without refinement.

Theorems & Definitions (2)

• Definition 1
• Definition 2