Directional Reasoning Injection for Fine-Tuning MLLMs

TL;DR

The paper tackles the gap in reasoning capabilities of multimodal LLMs (MLLMs) by showing that naive parameter-space merging is fragile and often unstable. It introduces DRIFT, a gradient-space directional prior that leverages the difference between a text-only reasoning expert and a multimodal backbone to bias multimodal fine-tuning without adding parameters. DRIFT demonstrates consistent reasoning gains across benchmarks like MathVista, MathVerse, MathVision, and WeMath while requiring only about 4K multimodal reasoning examples and roughly two hours of training, significantly reducing data and compute needs compared to training-heavy methods. This approach provides a scalable, data-efficient pathway for cross-domain reasoning transfer in MLLMs, balancing perception and reasoning with minimal overhead.

Abstract

Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts. Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning, both of which are resource-intensive. A promising alternative is model merging, which interpolates parameters between reasoning-enhanced LLMs and multimodal variants. However, our analysis shows that naive merging is not always a "free lunch": its effectiveness varies drastically across model families, with some (e.g., LLaVA, Idefics) benefiting while others (e.g., Qwen) suffer performance degradation. To address this, we propose Directional Reasoning Injection for Fine-Tuning (DRIFT) MLLMs, a lightweight method that transfers reasoning knowledge in the gradient space, without destabilizing multimodal alignment. DRIFT precomputes a reasoning prior as the parameter-space difference between reasoning and multimodal variants, then uses it to bias gradients during multimodal fine-tuning. This approach preserves the simplicity of standard supervised fine-tuning pipelines while enabling efficient reasoning transfer. Extensive experiments on multimodal reasoning benchmarks, including MathVista and MathVerse, demonstrate that DRIFT consistently improves reasoning performance over naive merging and supervised fine-tuning, while matching or surpassing training-heavy methods at a fraction of the cost.

Figures (4)

• Figure 1: DRIFT enables efficient reasoning transfer for MLLMs.Left: Compared with reasoning-oriented training methods, DRIFT achieves comparable performance while requiring dramatically less multimodal SFT data (4K vs. $>$59K examples). Right: Simple parameter merging performs poorly on multimodal reasoning benchmarks. Training-based methods improve performance but rely on costly data curation and multi-day training. In contrast, DRIFT reaches competitive results within $\sim$2 hours of training, making it both data- and compute-efficient.
• Figure 2: Layer/Module-wise analysis of model merging pairs. We compare LLaVA-Next-8B vs. Dart-Uniform, Idefics-8B vs. MetaMath, Qwen2-VL-7B vs. Qwen2-Math-7B, and Qwen2.5-VL-7B vs. DeepSeek-R1-Qwen-7B. Top Left: per-layer $\mathcal{L}_2$ norm differences. Bottom Left: per-layer cosine similarity. Top Right: average $\mathcal{L}_2$ norm differences for FFN layers and normalization layers. Bottom Right: average $\mathcal{L}_2$ norm differences for attention projections (Q/K/V/O).
• Figure 3: Overview of Directional Reasoning Injection (DRIFT). (a) Standard fine-tuning of a multimodal LLM $\phi_{VL}$, where gradients $g$ are applied directly to update trainable modules. (b) DRIFT modifies gradients by injecting a reasoning prior: $\tilde{g} = g + \alpha \cdot \text{scale}(g,\Delta)$, where $\Delta$ encodes the reasoning direction and $\text{scale}(\cdot)$ adjusts how $\Delta$ interacts with $g$. (c) The reasoning prior $\Delta$ is constructed as the parameter difference between a text-only reasoning model $\phi_{\text{reason}}$ and the multimodal variant $\phi_{VL}$. Our method enables reasoning knowledge to be transferred without destabilizing parameter-space merging.
• Figure 4: Qualitative example. DRIFT corrects a failure mode where the model’s visual perception is accurate but the reasoning chain leads to an incorrect answer.