Multimodal Language Models See Better When They Look Shallower

TL;DR

This work interrogates which ViT layers provide the most useful visual features for Multimodal LLMs, showing that shallow and middle layers can outperform deep ones on fine-grained tasks while deep layers excel at OCR and high-level semantics. It introduces Layer-wise Representation Similarity to partition ViT layers into three groups (shallow, middle, deep) and demonstrates a lightweight fusion method that concatenates multi-layer features and folds them through a single linear projection, achieving robust improvements across 60+ tasks and multiple model/data scales. The penultimate deep layer proves consistently strong, but middle layers (e.g., layer 18) offer notable advantages on counting and localization tasks, and a simple three-layer fusion (layers 23, 18, 3) often outperforms ad-hoc fusion baselines. The findings provide a principled understanding of visual layer selection in MLLMs and offer a practical, low-overhead fusion strategy to enhance multimodal reasoning without substantial architectural changes.

Abstract

Multimodal large language models (MLLMs) typically extract visual features from the final layers of a pretrained Vision Transformer (ViT). This widespread deep-layer bias, however, is largely driven by empirical convention rather than principled analysis. While prior studies suggest that different ViT layers capture different types of information, with shallower layers focusing on fine visual details and deeper layers aligning more closely with textual semantics, the impact of this variation on MLLM performance remains underexplored. We present the first comprehensive study of visual layer selection for MLLMs, analyzing representation similarity across ViT layers to establish shallow, middle, and deep layer groupings. Through extensive evaluation of MLLMs (1.4B-7B parameters) across 10 benchmarks encompassing 60+ tasks, we find that while deep layers excel in semantic-rich tasks like OCR, shallow and middle layers significantly outperform them on fine-grained visual tasks including counting, positioning, and object localization. Building on these insights, we propose a lightweight feature fusion method that strategically incorporates shallower layers, achieving consistent improvements over both single-layer and specialized fusion baselines. Our work offers the first principled study of visual layer selection in MLLMs, showing that MLLMs can often see better when they look shallower.

Figures (9)

• Figure 1: (a) Average cosine similarity of visual representations across different layers in CLIP-ViT. (b) Layer-wise performance on OCR tasks. The results highlight three distinct representation regions and their influence on performance.
• Figure 2: Averaged performance of layers 1 to 24 across various tasks. $\mathbf{General}$ represents tasks from MME, MMBench, GQA, and SEEDBench. $\mathbf{OCR}$ includes includes TextVQA and OCRBench. $\mathbf{CVB}$ corresponds to CVBench, whereas $\mathbf{VC}^{*}$ includes RefCOCO, RealWorldQA, and MMVet. Results show that the final layer underperforms the penultimate layer, and middle layers sometimes surpass deeper ones.
• Figure 3: Layer-wise performance distribution across four benchmarks: (a) MME, (b) MMVet, (c) MMBench, and (d) SEEDBench. The x-axis corresponds to layer indices and the y-axis indicates the sub-tasks index (see Tab. \ref{['appendix:subtasks2index']} for details). Top-performing layers for each sub-task are highlighted with color-coded markers: $\bullet$ (1st place), $\bullet$ (2nd place), and $\bullet$ (3rd place).
• Figure 4: Radar charts comparing the performance of Layers 23 and 24 across four different tasks under three LLM scales: 1.4B, 2.7B, and 7B. The results consistently show that the penultimate layer outperforms the final layer in all tasks. This trend remains stable across different model scales.
• Figure 5: Proportion of subtasks achieving their best performance at the penultimate layer on MME and SEEDBench, demonstrating a clear upward trend.