Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives

TL;DR

This work addresses catastrophic forgetting in multimodal large language models when faced with dynamic, cross-scenario visual data. It introduces MSVQA, a dataset spanning four scenarios, and UNIFIER, which decouples scene-specific visual representations into per-scenario branches and enforces cross-scenario consistency via a dedicated loss. Across 5- and 20-step continual learning benchmarks, UNIFIER outperforms baselines on VQA and visual grounding metrics and demonstrates knowledge accumulation within scenarios. The approach offers a practical path toward robust on-device multimodal continual reasoning in real-world data streams.

Abstract

Continual learning in visual understanding aims to deal with catastrophic forgetting in Multimodal Large Language Models (MLLMs). MLLMs deployed on devices have to continuously adapt to dynamic scenarios in downstream tasks, such as variations in background and perspective, to effectively perform complex visual tasks. To this end, we construct a multimodal visual understanding dataset (MSVQA) encompassing four different scenarios and perspectives including high altitude, underwater, low altitude and indoor, to investigate the catastrophic forgetting in MLLMs under the dynamics of scenario shifts in real-world data streams. Furthermore, we propose mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives (UNIFIER) to address visual discrepancies while learning different scenarios. Specifically, it decouples the visual information from different scenarios into distinct branches within each vision block and projects them into the same feature space. A consistency constraint is imposed on the features of each branch to maintain the stability of visual representations across scenarios. Extensive experiments on the MSVQA dataset demonstrate that UNIFIER effectively alleviates forgetting of cross-scenario tasks and achieves knowledge accumulation within the same scenario.

Figures (16)

• Figure 1: Task comparison and vision forgetting. Classical settings focus on parsing of user intent from text. In real visual tasks, the image background is more complex and the task is more challenging. Untrained model cannot precisely predict the bounding bbox while finetuning with the corresponding scenario can address the issue. But learning a new scenario results in vision forgetting (Severe false positives and false negatives).
• Figure 2: Examples of some image-text pair in the MSVQA dataset. More details are provided in the supplementary materials.
• Figure 3: The overview of Unifier. The left denotes a standard structure of a VLM. The middle denotes the details of the vision block in Unifier. The right denotes CSR module, which is added only on the vision encoder to isolate parameters for different scenarios.
• Figure 4: Training illustration of CSR module. Only one branch is involved in the training for a new scenario. The VCC loss constraint ensures consistency in the representations across various scenarios.
• Figure 5: Incremental trends on 10 steps setting in different scenarios. The performance gap is annotated at the end of each curve.