Low-rank Prompt Interaction for Continual Vision-Language Retrieval

TL;DR

This work tackles continual learning for vision–language retrieval by explicitly modeling cross-modal and cross-task interactions with a parameter-efficient framework. The proposed Low-rank Prompt Interaction (LPI) combines a shared low-rank decomposition for cross-modal prompts, hierarchical prompt alignment across layers, cross-modal prompt fusion, and cross-task prompt alignment driven by task semantic distances, trained with a composite loss that balances base task performance, cross-modal consistency, and cross-task discrimination. Key contributions include the Low-rank Interaction Decomposition (LID) to compress prompts, Hierarchical Prompt Alignment (HPA) to link prompts across layers, Cross-modal Prompt Fusion (CPF) for inter-layer collaboration, and Cross-task Prompt Alignment (CPA) leveraging task semantics; together they achieve superior recall with minimal parameter growth on image-text retrieval and referring expression comprehension under class-incremental settings. Empirical results show consistent gains over strong baselines and robust ablations confirm the value of each component, with analyses on computational cost and prompt visualization supporting practical viability and interpretability. The work advances efficient multimodal continual learning and offers a scalable approach for deploying vision–language models in evolving task streams; code is available at the provided repository.

Abstract

Research on continual learning in multi-modal tasks has been receiving increasing attention. However, most existing work overlooks the explicit cross-modal and cross-task interactions. In this paper, we innovatively propose the Low-rank Prompt Interaction (LPI) to address this general problem of multi-modal understanding, which considers both cross-modal and cross-task interactions. Specifically, as for the former, we employ multi-modal correlation modules for corresponding Transformer layers. Considering that the training parameters scale to the number of layers and tasks, we propose low-rank interaction-augmented decomposition to avoid memory explosion while enhancing the cross-modal association through sharing and separating common-specific low-rank factors. In addition, due to the multi-modal semantic differences carried by the low-rank initialization, we adopt hierarchical low-rank contrastive learning to ensure training robustness. As for the latter, we initially employ a visual analysis and identify that different tasks have clear distinctions in proximity. Therefore, we introduce explicit task contrastive constraints in the prompt learning process based on task semantic distances. Experiments on two retrieval tasks show performance improvements with the introduction of a minimal number of parameters, demonstrating the effectiveness of our method. Code is available at https://github.com/Kelvin-ywc/LPI.

Figures (7)

• Figure 1: Three Paradigms of Prompt Tuning in the Continual Learning Scenario.
• Figure 2: Illustration of the proposed LPI method. LPI consists of two parts, the cross-modal and cross-task interactions. As for the cross-modal interaction, we introduce three modules, low-rank interaction decomposition (LID), hierarchical prompt alignment (HPA), and cross-modal prompt fusion (CPF). LID decomposes high-dimensional prompts into low-rank vectors. HPA utilizes contrastive learning to align prompts across different layers. CPF achieves cross-layer information fusion between modalities. As for the cross-modal interaction, we introduce the cross-task prompt alignment (CPA) module which aligns prompts with task semantic distance. For each task, our method learns unique visual and textual prompts and interact modules. We store the prompts in the visual and textual prompt pool with task-specific keys which are used for predicting task identity during the inference period. As the task identity is unknown during the inference stage, we employ a method similar to that used in S-Prompts sprompts to predict the task identity.
• Figure 3: Interaction Module.
• Figure 4: Task Distance Visualization using t-SNE t-SNE.
• Figure 5: Metrics with Different Prompt and Interaction Rank.