Multimodal Instruction Tuning with Conditional Mixture of LoRA

TL;DR

This work addresses the problem of task interference in parameter-efficient multimodal instruction tuning by introducing MixLoRA, a dynamic, input-conditioned extension of LoRA. MixLoRA constructs per-input adaptation matrices $\Delta W = BA = \sum_{i=1}^r b_i \otimes a_i$ from large pools of factors using two independent routers for $A$ and $B$ and a conditional router that ties $B$ to $A$, enabling robust zero-shot generalization across diverse multimodal tasks. Empirical results on Vision-Flan training and evaluation on MME plus seven multimodal benchmarks show that MixLoRA consistently outperforms standard LoRA at the same rank and can surpass higher-rank LoRA, while analyses of routing strategies and unseen tasks reveal effective factor activation and reduced task interference. The approach provides a scalable, parameter-efficient path toward robust multimodal generalization and highlights the value of instance-based dynamic routing in PEFT for complex, cross-modal tasks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pre-trained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameter-efficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks.

Figures (7)

• Figure 1: Comparative Overview of LoRA and MixLoRA.Left: The conventional LoRA with static low-rank decomposition matrices $BA$. Right: MixLoRA treats the low-rank decomposition factors as experts that can be selectively combined through a Dynamic Factor Selection module, enabling the construction of varied low-rank decomposition matrices $A$ and $B$ tailored to varying input scenarios. The selected factors are visually distinguished by color coding: green for $B$ and blue for $A$.
• Figure 2: The Task Interference Score $\mathcal{I}$ for LoRA decomposition matrices $A$ and $B$. Each cell in the heatmap corresponds to the average interference score $\mathcal{I}_{i,j}$ of task $j$ (column) on the task $i$ (row). A blue hue indicates a negative impact of task $j$ on task $i$, whereas a red hue signifies a positive impact.
• Figure 3: Dynamic Factor Selection in MixLoRA. MixLoRA treats low-rank decomposition factors as experts and dynamically constructs the LoRA $A$ and $B$ through two independent routers $R_{\text{IFS}}^A(\cdot)$ and $R_{\text{IFS}}^B(\cdot)$, complemented by a conditional router $R^B_{\text{CFS}}(\cdot)$.
• Figure 4: Effect of Conditional Factor Selection
• Figure 5: T-SNE Visualization of Factor Selection Distribution for MixLoRA ($E$ = 32, $r$ = 8). Instances are represented as points, where instances from the same task share a common color.