CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering

TL;DR

CluMo tackles catastrophic forgetting in continual vision-language VQA by learning modality-specific prompt keys through mini-batch K-means and combining them to select fusion prompts. The approach uses a two-stage training regime that first optimizes visual and textual prompt keys and then freezes them while learning task-specific prompts, augmented by a knowledge-distillation loss to stabilize updates. Empirical results on CLOVE-scene and CLOVE-function show state-of-the-art performance against regularization, rehearsal, and prior prompt-based CL methods, highlighting strong generalization and memory efficiency without replay. This work provides a practical, scalable path for multimodal continual learning in vision-language tasks.

Abstract

Large vision-language models (VLMs) have shown significant performance boost in various application domains. However, adopting them to deal with several sequentially encountered tasks has been challenging because finetuning a VLM on a task normally leads to reducing its generalization power and the capacity of learning new tasks as well as causing catastrophic forgetting on previously learned tasks. Enabling using VLMs in multimodal continual learning (CL) settings can help to address such scenarios. To improve generalization capacity and prevent catastrophic forgetting, we propose a novel prompt-based CL method for VLMs, namely $\textbf{Clu}$ster-based $\textbf{Mo}$dality Fusion Prompt (\textbf{CluMo}). We design a novel \textbf{Key-Key-Prompt} pair, where each prompt is associated with a visual prompt key and a textual prompt key. We adopt a two-stage training strategy. During the first stage, the single-modal keys are trained via $K$-means clustering algorithm to help select the best semantically matched prompt. During the second stage, the prompt keys are frozen, the selected prompt is attached to the input for training the VLM in the CL scenario. Experiments on two benchmarks demonstrate that our method achieves SOTA performance.

Figures (8)

• Figure 1: Comparison between existing prompt-based CL methods and our proposed method: $\textbf{(a)}$ Uni-modal based methods use image feature to select prompts from a prompt pool. $\textbf{(b)}$ Multi-modal based methods use the image features to select image prompts and use the text features to select the text prompts. $\textbf{(c)}$ We first train the prompt key using a clustering algorithm to form a cluster key and use the combination of the cluster key from both modalities to select the fusion prompt.
• Figure 2: Block diagram of the proposed approach: $\textbf{Left}$: the backbone contains a pre-trained frozen visual encoder, a textual encoder, and a multimodal encoder. The answer decoder shares the same architecture as multimodal encoder. During the training phase, the fixed number of visual prompt keys, textual prompt keys, and a prompt pool will be added for each new task. $\textbf{Right}$: the procedure of visual prompt key training consists of training the modal-specific prompt key by a sequence of randomly selected batches of training data from current task until convergence is reached. Same procedure for textual prompt key.
• Figure 3: Naive Example of Prompt Selection. Consider a naive animal VQA dataset which only contains dog and cat images and questions only about "what" and "where". During the first stage training, the visual prompt keys and textual prompt keys are learnt to represent "dog"/"cat" and "where"/"what" respectively (in realistic settings the keys will learn the implicit cluster instead of explicit category). Given a test image-question pair, the image and question are projected to their modality-specific feature space through the encoders. The nearest prompt keys, which are keys represent "dog" and "what" are selected. The combination of the two prompt keys lead to "prompt3" which is finally selected.
• Figure 4: Cluster distribution on all training image data of $\textbf{CLOVE-Scene}$'s six sub-tasks before and after applying mini-batch $k$-means clustering algorithm with visual key size of 3 using PCA. Color of more diversity indicates more even distribution of key selection.
• Figure 5: Cluster distribution on all training text data of $\textbf{CLOVE-Scene}$'s six sub-tasks before and after applying mini-batch $k$-means clustering algorithm with textual key size of 3 using PCA.