Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference

TL;DR

This work reveals a depth-dependent shift in how MLLMs process visual information: shallow layers primarily inject image information into instruction tokens, while deeper layers rely on intra-visual, token-token interactions to refine visual representations. Building on this insight, the authors introduce Hierarchical Modality-Aware Pruning (HiMAP), a plug-and-play inference acceleration method that dynamically prunes image tokens at strategically chosen layers, achieving substantial FLOPs reductions (over 65%) and about 50–60% faster inference with minimal impact on accuracy. The approach is validated across diverse vision-language tasks and model scales, showing consistent speedups and often improved or preserved performance compared to baselines like FastV. The work also provides a detailed diagnostic framework, including saliency-based modality impact analyses and bias-ratio metrics, to understand the evolving visual information flow in MLLMs. Overall, HiMAP offers a practical, theory-backed route to efficient inference in multimodal large language models with broader implications for real-time multimodal AI systems.

Abstract

Multimodal large language models (MLLMs) improve performance on vision-language tasks by integrating visual features from pre-trained vision encoders into large language models (LLMs). However, how MLLMs process and utilize visual information remains unclear. In this paper, a shift in the dominant flow of visual information is uncovered: (1) in shallow layers, strong interactions are observed between image tokens and instruction tokens, where most visual information is injected into instruction tokens to form cross-modal semantic representations; (2) in deeper layers, image tokens primarily interact with each other, aggregating the remaining visual information to optimize semantic representations within visual modality. Based on these insights, we propose Hierarchical Modality-Aware Pruning (HiMAP), a plug-and-play inference acceleration method that dynamically prunes image tokens at specific layers, reducing computational costs by approximately 65% without sacrificing performance. Our findings offer a new understanding of visual information processing in MLLMs and provide a state-of-the-art solution for efficient inference.

Figures (13)

• Figure 1: Illustration of our hypothesis. In shallow layers, image tokens inject most of the visual information into instruction tokens, establishing a cross-modal semantic representation for subsequent computations. In deeper layers, image tokens aggregate the residual visual information, refining the semantic representation within the visual modality.
• Figure 2: Distribution of Input Sequence Tokens. Image tokens constitute 77% of the total input tokens, nearly double the combined total of system and instruction tokens. This highlights a considerable computational overhead associated with image tokens.
• Figure 3: Contributions of different modalities to prediction outcomes across layers. The contribution of visual modality is significantly lower than textual modality.
• Figure 4: Importance of intra-visual flow and visual-textual flow across layers. Dominant flow of visual information shifts as model depth increases.
• Figure 5: Disrupting visual-textual flow versus disrupting visual-random flow within the first or last 5 layers. Disrupting visual-textual flow in the first 5 layers has the most substantial effect, highlighting shallow-layers information injection from image tokens to instruction tokens.