Is Parameter Isolation Better for Prompt-Based Continual Learning?

TL;DR

The paper challenges fixed, per-task prompt isolation in prompt-based continual learning and proposes Hash, a shared MoE-inspired prompt pool with dynamic routing and a history-aware modulator. The approach enables flexible, input-driven prompt composition while protecting frequently used prompts from excessive updates, addressing both parameter efficiency and catastrophic forgetting. Empirical results across diverse benchmarks (CIFAR-100, ImageNet-R, CUB-200, and 5-Datasets) show Hash achieving state-of-the-art FAA and CAA with competitive forgetting, and ablations confirm the complementary benefits of the history-aware routing and gradient modulation components. The work demonstrates that parameter sharing with careful history-aware control yields robust, scalable continual learning in both class- and domain-incremental settings, with favorable training costs.

Abstract

Prompt-based continual learning methods effectively mitigate catastrophic forgetting. However, most existing methods assign a fixed set of prompts to each task, completely isolating knowledge across tasks and resulting in suboptimal parameter utilization. To address this, we consider the practical needs of continual learning and propose a prompt-sharing framework. This framework constructs a global prompt pool and introduces a task-aware gated routing mechanism that sparsely activates a subset of prompts to achieve dynamic decoupling and collaborative optimization of task-specific feature representations. Furthermore, we introduce a history-aware modulator that leverages cumulative prompt activation statistics to protect frequently used prompts from excessive updates, thereby mitigating inefficient parameter usage and knowledge forgetting. Extensive analysis and empirical results demonstrate that our approach consistently outperforms existing static allocation strategies in effectiveness and efficiency.

Figures (5)

• Figure 1: (a) Accuracy of novel classes at each incremental step on the CUB200 dataset. Our method achieves better novel class learning ability compared to static prompt allocation. (b) Comparison of parameter count and accuracy on the Imagenet-R dataset. Our method demonstrates a better accuracy–parameter trade-off than existing approaches. (c) Comparison of expert activation frequency distribution on the Imagenet-R dataset. Our proposed history-aware modulator achieves superior load balancing while effectively reducing knowledge forgetting (FR).
• Figure 2: Network architecture overview: Our method adopts a MoE-based shared prompt strategy. During training, the historical activation count of each prompt is recorded and incorporated as a bias term into the router’s scoring mechanism, enabling dynamic and informed expert selection based on historical usage.
• Figure 3: (a) Forgetting rate reduction with history-aware modulator. (b) Cross-task expert utilization frequency distribution without history-aware modulator. (c) Cross-task expert utilization frequency distribution with history-aware modulator.
• Figure 4: Long sequence incremental analysis: (a) Test accuracy after every 10 sessions in the 50-session setting on Split CIFAR-100, comparing Hash and CODA. (b) Test accuracy across different total session numbers.
• Figure 5: Ablation study on the hyperparameter of history-aware module.