SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning

TL;DR

SEFE tackles forgetting in Multimodal Continual Instruction Tuning by disentangling superficial forgetting from essential forgetting. It introduces Answer Style Diversification (ASD) to mitigate formatting bias and CoIN-ASD as a standardized benchmark, and RegLoRA to stabilize prior knowledge during continual learning. ASD unifies answer formats by transforming task data into five styles, while RegLoRA focuses regularization on the most influential LoRA updates to preserve knowledge. Across CoIN and CoIN-ASD, SEFE achieves state-of-the-art results, validating the complementary roles of ASD and RegLoRA in reducing both forgetting types and improving knowledge retention.

Abstract

Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting. In this paper, we explore forgetting in this context, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model's knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks' answer styles, making the results unusable. By contrast, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can obscure the model's knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for transforming data styles across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization, enabling the model to retain existing competencies. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.

Figures (7)

• Figure 1: Examples illustrating superficial forgetting and essential forgetting. (a) Instruction; (b) Response case without forgetting; (c) and (d) Response cases with superficial forgetting; (e) Response case with essential forgetting.
• Figure 2: An example of the ASD paradigm applied to a dataset consisting solely of short answer questions. Through the ASD process, $(100 - X)$% of samples are retained in their original form, while the remaining $X$% are equally transformed into four alternative formats: yes/no, multiple-choice, brief explanation, and detailed explanation. Additional examples are provided in Appendix \ref{['sec:asd_detail']}.
• Figure 3: Overview of RegLoRA. In each past LoRA, large values in the weight update matrix are identified as key elements. When training a new LoRA, these key positions are incorporated into a regularization mask to enforce targeted constraints.
• Figure 4: Case studies of main components in the proposed SEFE method. (a) Instruction; (b) Response from the baseline model (LoRA); (c) Response from the baseline model with ASD added; (d) Response from the baseline model with both ASD and RegLoRA added; (e) Basic information of the case. Additional case studies can be found in Appendix \ref{['sec:add_cs']}.
• Figure 5: Specific Transformation Rules of ASD.