Dynamic Routing Between Experts: A Data-Efficient Approach to Continual Learning in Vision-Language Models

TL;DR

This work tackles catastrophic forgetting in vision-language models (VLMs) during sequential task fine-tuning by introducing a routing-based continual learning framework. It leverages task-specific LoRA adapters and token-level routing vectors that gate expert contributions, allowing new tasks to be integrated with data from the current task only while preserving the model's foundational capabilities. Empirical results on InternVL-2 models (2B and 8B) show routing achieves competitive specialized-task performance relative to multi-task learning and mitigates forgetting on benchmark tasks like ChartQA, MMBench, and DocVQA, with additional benefits in cross-modal transfer that improve performance across modalities. The approach scales with model size, reduces data and compute overhead, and offers a practical path toward scalable continual learning in large vision-language systems, with future potential for cross-lingual extensions.

Abstract

Vision-Language Models (VLMs) suffer from catastrophic forgetting when sequentially fine-tuned on new tasks, degrading performance on previously learned foundational and task-specific capabilities. While multi-task learning can mitigate forgetting, it requires simultaneous access to all datasets and imposes computational overhead that scales linearly with the number of tasks. In this work, we introduce a routing-based approach that enables the integration of new tasks while preserving the foundational knowledge acquired during pretraining. We evaluate our method using InternVL-2 models (2B and 8B parameters) and demonstrate that routing preserves the model's foundational capabilities by maintaining performance on general-purpose benchmarks such as ChartQA, MMBench, and DocVQA, while simultaneously improving accuracy on specialized tasks. Importantly, our approach achieves this without requiring concurrent access to data from all tasks, avoiding the significant computational and data overhead associated with traditional multi-task learning. We further conduct extensive ablation studies to evaluate the scalability and robustness of routing-based learning, showing that the approach is resilient to a growing number of tasks and performs particularly well when new tasks are semantically related. Finally, we show that the routing mechanism enables superior cross-modal transfer between language and vision capabilities, allowing knowledge learned in one modality to enhance performance in another capability not achieved by existing continual learning methods.

Figures (4)

• Figure 1: Comparison of Multi-Task Learning, Routing-Based Methods and Sequential FT in mitigating catastrophic forgetting. Routing-based methods match multi-task learning performance while requiring only new task data, avoiding costly retraining on previous tasks. Tasks are introduced sequentially from left to right, with the multi-task model trained on the cumulative data from all prior tasks. In contrast, sequential fine-tuning only sees new task data, leading to strong performance on the most recent task but potential forgetting of earlier ones.
• Figure 2: Performance drop in comparison to specialized model for 2B and 8B models.
• Figure 3: Routing patterns for SNLI (left), MMBENCH (middle), and COCO (right). The figure demonstrates that the routing model correctly activates task-specific modules: SNLI and XNLI for SNLI, only COCO for COCO, and no module for MMBENCH, where the base model handles the task. These confirm the routing mechanism's effectiveness in selecting the appropriate module based on the task.
• Figure 4: Routing patterns for the MGSM dataset in multilingual transfer. Notably, the model leverages the Chinese expert in the early and later stages, while the math expert is primarily used for computation.