Multiple Queries with Multiple Keys: A Precise Prompt Matching Paradigm for Prompt-based Continual Learning

TL;DR

This work tackles prompt selection bias in prompt-based continual learning by introducing MQMK, a paradigm that uses Multiple Queries for task-level breadth and Multiple Keys for fine-grained, class-level depth. The method employs a local matching mechanism to align test queries with task-specific keys, optimizing both classification and prompt–distribution alignment with a combined loss $\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{mch}$. An efficient inference variant, MQMK-EI, fuses knowledge across tasks to maintain speed while preserving accuracy, and the framework views prompts as a coordinated ensemble across tasks rather than per-task voting. Empirical results on CIFAR-100, ImageNet-R, and DomainNet show MQMK achieves state-of-the-art performance with substantial gains in prompt matching rate (over $30\%$) and competitive efficiency, validating the proposed local matching approach and the benefit of task-aware prompt design. The work also discusses practical trade-offs between accuracy, memory, and computation, and provides analyses on the forgetting behavior under different prompt fusion strategies.

Abstract

Continual learning requires machine learning models to continuously acquire new knowledge in dynamic environments while avoiding the forgetting of previous knowledge. Prompt-based continual learning methods effectively address the issue of catastrophic forgetting through prompt expansion and selection. However, existing approaches often suffer from low accuracy in prompt selection, which can result in the model receiving biased knowledge and making biased predictions. To address this issue, we propose the Multiple Queries with Multiple Keys (MQMK) prompt matching paradigm for precise prompt selection. The goal of MQMK is to select the prompts whose training data distribution most closely matches that of the test sample. Specifically, Multiple Queries enable precise breadth search by introducing task-specific knowledge, while Multiple Keys perform deep search by representing the feature distribution of training samples at a fine-grained level. Each query is designed to perform local matching with a designated task to reduce interference across queries. Experiments show that MQMK enhances the prompt matching rate by over 30\% in challenging scenarios and achieves state-of-the-art performance on three widely adopted continual learning benchmarks. The code is available at https://github.com/DunweiTu/MQMK.

Figures (10)

• Figure 1: (a) Prompts inconsistent with test samples may introduce bias. (b) Prior prompt selector uses a single query and task-level keys; our method adopts task-level breadth queries and class-level depth keys, involving prompts in the selection process.
• Figure 2: The average accuracy for four scenarios: when SQSK selects False Prompt and True Prompt, when samples initially with False Prompt are manually replaced with True Prompt, and when all samples use True Prompt (Perfect Match).
• Figure 3: Overall pipeline of MQMK. Training: Select the true prompt based on the label information and input it into hidden states to obtain the corresponding query for classification. This process updates the prompt, the locally matched key and the classifier. Inference: Feed all prompts into hidden states to generate the query pool and select the most consistent query for classification by locally matching the task-level queries with the corresponding class-level keys.
• Figure 4: Ablation experiments on the 10-task Split ImageNet-R. The average accuracy and matching rate change with the learning process.
• Figure 5: Average accuracy of MQMK and SQSK on the 10-task Split ImageNet-R under different depths and lengths of the e-prompt. (a) The length is fixed at 80, and the depth varies. (b) The depth is fixed at 10, and the length varies.