PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

TL;DR

This work targets catastrophic forgetting in continual instruction tuning for multimodal LLMs by addressing Misaligned Co-drift between routing decisions and expert updates in MoE-LoRA. It introduces Pathway Activation Subspaces (PASs), defined as $\mathcal{S}_e=\mathrm{span}(A_e^\top)$, to tie each expert's low-rank pathway to its routing and preservation signals. The method comprises PASs-Guided Reweighting (PASs-RW), which derives routing from expert activation signals, and PASs-Aware Rank Stabilization (PASs-RS), which stabilizes rank directions important to prior tasks. Experiments on MLLM-CTBench show consistent gains in final performance (AP) and reduced forgetting (BWT) over strong baselines and MoE-LoRA variants, without additional parameters, demonstrating the practical value of PASs for CIT.

Abstract

Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input-expert specialization. We term this phenomenon Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting.To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE-LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE-LoRA variants in both accuracy and anti-forgetting without adding parameters. Our code will be released upon acceptance.

Figures (7)

• Figure 1: Illustration of the Misaligned Co-drift in existing MoE-LoRA methods, where the thickness of arrow indicates the sampling probability of router. Here, the drift in router assignments and the internal drift within expert parameters for the same task jointly drive this issue, thereby exacerbating catastrophic forgetting.
• Figure 2: Overview of the proposed PASs-based MoE-LoRA method. We consider continual instruction tuning with a fixed set of $E$ LoRA experts. (Bottom) Each input $x^t$ generates a low-rank response $z_e$ and activation energy $s_e$ via the Pathway Activation Subspace (PASs) of each expert. (Right) PASs-guided Reweighting (PASs-RW) leverages $s_e$ to compute mixture weights, eliminating the need for an independent router. (Top) PASs-aware Rank Stabilization (PASs-RS) tracks rank-level importance $I^{\mathrm{agg}}$ (represented by bar heights) and applies stabilization constraints (indicated by lock icons) to protect historically critical directions from drift.
• Figure 3: Comparison of final performance and router stability. Top and bottom panels display the final accuracy and the corresponding router stability across the task stream, respectively. Stability is quantified as the negative Mean Jensen--Shannon (JS) divergence.
• Figure 4: Ablation results on the regularization strength $\lambda$ for PASs-MoE in MLLM-CTBench. The upper panel reports AP, while the lower panel shows BWT.
• Figure 5: Update magnitude versus old-task importance across representative layers. Each plot bins coefficients by the previous task's aggregated importance $I_{\mathrm{agg}}$ (low $\rightarrow$ high) and reports the median (with interquartile range) of the log update norm $\log \|\Delta W\|$ within each bin.