CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering

TL;DR

This work tackles continual visual question answering (VQA) with multimodal large language models by addressing catastrophic forgetting in non-stationary data streams. It introduces CL-MoE, a dual-ring MoE framework combining Dual-Router MoE (instance-level and task-level routers) with Dynamic Momentum MoE (MMoE) on a LoRA-based MoE backbone, enabling robust knowledge absorption via momentum updates: $\hat{\mathbf{x}} = \beta \frac{\alpha}{r} \sum_{i=1}^n G^I(\mathbf{x})_i E_i(\mathbf{x}) + (1-\beta) \frac{\alpha}{r} \sum_{i=1}^n G^T(t)_i E_i(\mathbf{x})$ and $\theta_t = \mathbf{\lambda} \circ \theta_{t-1} + (\mathbf{1}-\mathbf{\lambda}) \circ \varphi_t$, where $\mathbf{\lambda}$ depends on expert roles. The approach yields state-of-the-art results on VQA v2 across 10 subtasks, with strong forward and backward transfer and ablations confirming the value of both routing components and momentum updates. The framework demonstrates practical impact by enabling continual multimodal reasoning without full retraining, leveraging rich commonsense knowledge in LLMs. Overall, CL-MoE advances continual learning for MLLMs by combining local/global expert routing with momentum-based consolidation to mitigate forgetting while enhancing knowledge integration.

Abstract

Multimodal large language models (MLLMs) have garnered widespread attention from researchers due to their remarkable understanding and generation capabilities in visual language tasks (e.g., visual question answering). However, the rapid pace of knowledge updates in the real world makes offline training of MLLMs costly, and when faced with non-stationary data streams, MLLMs suffer from catastrophic forgetting during learning. In this paper, we propose an MLLMs-based dual momentum Mixture-of-Experts (CL-MoE) framework for continual visual question answering (VQA). We integrate MLLMs with continual learning to utilize the rich commonsense knowledge in LLMs. We introduce a Dual-Router MoE (RMoE) strategy to select the global and local experts using task-level and instance-level routers, to robustly assign weights to the experts most appropriate for the task. Then, we design a dynamic Momentum MoE (MMoE) to update the parameters of experts dynamically based on the relationships between the experts and tasks/instances, so that the model can absorb new knowledge while maintaining existing knowledge. The extensive experimental results indicate that our method achieves state-of-the-art performance on 10 VQA tasks, proving the effectiveness of our approach.

Figures (3)

• Figure 1: Progress of continual learning over time on VQA v2. We give the results of previous CL methods based on VL-T5 and LLaVA. Multitask represents the upper bound of the model, which trains over all the tasks once.
• Figure 2: The framework of our CL-MoE that contains Dual-Router MoE (RMoE) and Dynamic Momentum MoE (MMoE). We propose RMoE to capture the local and global experts using the instance-level and task-level routers. Then, MMoE dynamically updates the parameters of task-shared and task-specific experts selected by RMoE using a momentum strategy.
• Figure 3: Performance(%) of our CL-MoE with different hyperparameters $\gamma$ and $\beta$ on VQA v2.