Modality-Inconsistent Continual Learning of Multimodal Large Language Models

TL;DR

This paper defines Modality-Inconsistent Continual Learning (MICL) for Multimodal Large Language Models, where tasks introduce both modality shifts (image, audio, video) and varying task types (captioning, QA), causing catastrophic forgetting. It proposes MoInCL, comprising a Pseudo Target Generation Module (PTGM) to synthesize task-type targets within learned modalities and an Instruction-based Knowledge Distillation (IKD) constraint to preserve LLM capabilities across modalities. The approach is evaluated on six tasks spanning three modalities, showing significant improvements over representative and state-of-the-art continual learning baselines, with strong anti-forgetting performance and cross-modal retention. The work advances practical continual learning for MLLMs and highlights the importance of addressing both modality and task-type shifts for robust multimodal instruction-following systems.

Abstract

In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike existing vision-only or modality-incremental settings, MICL combines modality and task type shifts, both of which drive catastrophic forgetting. To address these challenges, we propose MoInCL, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities. It also incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced. We benchmark MICL using a total of six tasks and conduct experiments to validate the effectiveness of our proposed MoInCL. The experimental results highlight the superiority of MoInCL, showing significant improvements over representative and state-of-the-art continual learning baselines.

Figures (8)

• Figure 1: Illustration of our proposed Modality-Inconsistent Continual Learning (MICL), a novel and practical continual learning scenario of Multimodal Large Language Models (MLLMs), where tasks involve inconsistent modalities (image, video, or audio) and varying task types (captioning or question-answering).
• Figure 2: Overview of our proposed MoInCL, which mainly consists of a Multimodal Large Language Model (MLLM), a Pseudo Target Generation Module (PTGM), and a Instruction-based Knowledge Distillation (IKD). The red fire icon denotes the component is trainable in the current task, and the snowflake icon denotes the component is frozen during the training of the current task, while the blue fire icon means the associate component is trainable with LoRA hu2022lora when training on the current task.
• Figure 3: Qualitative results of the Fine-tuning method in Order 2. The sample is randomly selected from the test set of Task 1 (Image Captioning). The results are generated using models trained after after (a) Task 1, (b) Task 2, (c) Task 3, (d) Task 4, (e) Task 5, and (f) Task 6.
• Figure 4: Qualitative results of the LwF li2017learning method in Order 2. The sample is randomly selected from the test set of Task 1 (Image Captioning). The results are generated using models trained after after (a) Task 1, (b) Task 2, (c) Task 3, (d) Task 4, (e) Task 5, and (f) Task 6.
• Figure 5: Qualitative results of the EWC kirkpatrick2017overcoming method in Order 2. The sample is randomly selected from the test set of Task 1 (Image Captioning). The results are generated using models trained after after (a) Task 1, (b) Task 2, (c) Task 3, (d) Task 4, (e) Task 5, and (f) Task 6.