Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention

TL;DR

This work investigates how vision and language are fused in Multimodal LLMs by performing layer‑wise visual masking to locate fusion layers and a late re‑attention (the review layer) around layer $29$. It introduces a training‑free contrastive attention mechanism that compares post‑integrated attention $A^{(k)}$ with a pre‑integrated layer $A^{(i^*)}$ to highlight meaningful attention shifts and suppress high‑noise regions during the review stage. The method identifies key fusion layers (early fusion set $ ext{S}$ and the post‑integrated layer 28) and demonstrates substantial, training‑free performance gains across six VQA benchmarks with LLaVA variants, achieving state‑of‑the‑art results. These findings advance interpretability of MLLMs and offer a lightweight, effective strategy for enhancing multimodal reasoning without additional training.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language understanding, yet how they internally integrate visual and textual information remains poorly understood. To bridge this gap, we perform a systematic layer-wise masking analysis across multiple architectures, revealing how visual-text fusion evolves within MLLMs. The results show that fusion emerges at several specific layers rather than being uniformly distributed across the network, and certain models exhibit a late-stage "review" phenomenon where visual signals are reactivated before output generation. Besides, we further analyze layer-wise attention evolution and observe persistent high-attention noise on irrelevant regions, along with gradually increasing attention on text-aligned areas. Guided by these insights, we introduce a training-free contrastive attention framework that models the transformation between early fusion and final layers to highlight meaningful attention shifts. Extensive experiments across various MLLMs and benchmarks validate our analysis and demonstrate that the proposed approach improves multimodal reasoning performance. Code will be released.

Figures (7)

• Figure 1: Contrastive attention (pre-integrated $\rightarrow$ last layer) showing focus shift and persistent high-attention noise.
• Figure 2: Layer-wise masking effects on accuracy and inference time across multimodal datasets using LLaVA-1.5. Acc is the performance after completely masking the visual information at the corresponding layer.
• Figure 3: Layer-wise masking effects on accuracy and inference time across multimodal datasets using LLaVA-1.6.
• Figure 4: overview of contrastive attention and review-stage masking.
• Figure 5: Layer-wise Hellinger distance to the final layer.