Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging

TL;DR

The paper tackles transferring explicit reasoning ability from math-focused LLMs to Vision-Language Models via training-free cross-modal model merging. It introduces a linear merging framework based on task vectors and evaluates it across diverse VLMs and math-reasoning task vectors, showing robust gains on math benchmarks with limited impact on perception tasks. A key contribution is the interpretability analysis: masking modules and layer-wise probing reveal perception resides in early layers while reasoning emerges in middle-to-late layers, and merging distributes reasoning across more layers. The work demonstrates inference-time scaling benefits and provides actionable insights for multimodal integration and interpretability, with practical implications for improving multimodal reasoning without costly retraining.

Abstract

Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.

Figures (10)

• Figure 1: Illustration of our work investigating how model merging works when transferring reasoning ability from a Math-specific LLM to the VLM, showcasing the effects and results of the merged model, as well as the interpretation of layer-wise abilities. The abilities are represented in corresponding colors: red indicates perception ability hidden in early layers, while blue denotes reasoning ability hidden in relatively later layers.
• Figure 2: Accuracy changes after merging compared to the baseline. Generally, datasets directly requiring math-related and text-dominant capabilities, such as textbook QA and math word problem, exhibit clear improvements while domains requiring visual processing such as figure QA show performance degradation.
• Figure 3: The relationship between answer length change (characters) and accuracy improvement after merging (The x-axis represents the relative change from the original answer, while the y-axis shows the absolute change in accuracy), classified by task and skills.
• Figure 4: LLaVA → LLaMA ( and ): the accuracy changes after replacing the parameters of each MLP ( ) and Attention ( ) layer of the LLaVA model with parameters from LLaMA. We find that masking out the early layers has a greater impact on general VQA tasks than masking the later layers, suggesting that perception abilities gained from LLaVA-sft training are primarily located in the early layers. Dart-Merged LLaVA → LLaVA ( and ): the accuracy changes after replacing each MLP ( ) and Attention layer ( ) of the Dart-Merged LLaVA model to that of LLaVA. A significant drop in accuracy in math-targeted VQA tasks is observed from 5 layer onwards. (highlighted in blue), suggesting that the reasoning ability is mainly located on these layers.
• Figure 5: LLaVA → 1/N ( and ): the accuracy changes after replacing the parameters of each MLP ( ) and Attention ( ) layer of the LLaVA model with $\frac{1}{N}$, where $N$ is the first dimension of the weight matrix. The highlighted red area shows that early-to-middle layers are more crucial for both general and math-related tasks in the LLaVA model, as evidenced by the significant drops, i.e., 0.25 absolute accuracy drop in general tasks and 0.10 in math-targeted VQA. Dart-Merged LLaVA → 1/N ( and ): the accuracy changes after replacing the parameters of each MLP ( ) and Attention ( ) layer of the Dart-Merged LLaVA model with $\frac{1}{N}$. Comparing before and after merging when applied masking out, we observe a larger drop in accuracy in math-targeted VQA tasks across all layers (highlighted in blue), suggesting that the contribution of all most all layers to math reasoning has increased.