SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

TL;DR

This work tackles continual multimodal instruction tuning (MCIT) with mixture-of-experts by identifying two forgetting mechanisms: router drift and expert drift. It introduces StAbilized Mixture-of-Experts (SAME), combining spectral-aware routing to confine updates to task-relevant subspaces, curvature-aware scaling to precondition updates against historical input geometry, and adaptive expert activation to freeze underutilized or historically important experts during a task. These components jointly stabilize expert routing, preserve prior task functionality, and reduce cross-task interference without rehearsal. Empirical results on the CoIN benchmark show SAME achieving state-of-the-art final performance with notable improvements in long-horizon stability and training efficiency, validating its effectiveness in practical MCIT deployments. The approach offers a principled, scalable framework for preserving knowledge across sequential vision-language tasks while maintaining adaptability to new instructions.

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. Extensive experiments demonstrate its SOTA performance.

Figures (9)

• Figure 1: (a$\sim$c) On the Task 1 test set, the router’s expert-activation distribution shifts as new tasks are learned, with decreasing overlap against later-task routers, indicating router drift. (d) The left y-axis shows the normalized entropy, defined as the entropy divided by the maximum possible entropy over $n$ experts. Even after re-training the router on Task 1 while freezing experts from each stage, the recovered Task 1 accuracy drops across tasks and the routing entropy decreases, revealing expert drift beyond misrouting.
• Figure 2: Overview of Same. Same stabilizes MoE adaptation by (i) tracking the router-input covariance and performing spectral-aware routing updates in task-relevant subspaces, (ii) applying curvature-aware scaling to bound expert degradation under historical input geometry, and (iii) using adaptive expert activation to freeze selected experts during each task .
• Figure 3: Impact of spectral-aware routing. Adding the spectral-aware routing strategy enables more consistent expert selection.
• Figure 4: Impact of curvature-aware scaling. Adding curvature-aware scaling improves re-routing accuracy on Task 1, indicating stronger preservation of early-task expert functionality.
• Figure 5: Impact of adaptive expert activation on training efficiency. By freezing low-utility yet historically important experts during each task, our method reduces per-task training time and GPU memory footprint across continual instruction tuning tasks.