$\mathcal{V}isi\mathcal{P}runer$: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs

TL;DR

This work reveals that cross-modal processing in MLLMs is three-stage and discontinuous: shallow layers perform task recognition, middle layers sparsely fuse a small set of vision tokens, and deep layers linguistically refine the output after fusion. It introduces VisPruner, a training-free pruning framework that dramatically reduces vision-related attention (up to 99%) and FLOPs (≈53.9%) on LLaVA-v1.5 7B while maintaining performance, and demonstrates generalization across diverse MLLMs. The authors critique attention-based analyses as insufficient for determining visual token utility, proposing an influence-based, token-wise pruning approach and layer-wise strategies such as vision sinks and early-exit design. The findings offer actionable guidelines for designing efficient MLLMs and highlight practical steps to integrate these principles into future training and inference pipelines.

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance across vision-language tasks, but suffer from significant computational overhead due to the quadratic growth of attention computations with the number of multimodal tokens. Though efforts have been made to prune tokens in MLLMs, \textit{they lack a fundamental understanding of how MLLMs process and fuse multimodal information.} Through systematic analysis, we uncover a \textbf{three-stage} cross-modal interaction process: (1) Shallow layers recognize task intent, with visual tokens acting as passive attention sinks; (2) Cross-modal fusion occurs abruptly in middle layers, driven by a few critical visual tokens; (3) Deep layers discard vision tokens, focusing solely on linguistic refinement. Based on these findings, we propose \emph{VisiPruner}, a training-free pruning framework that reduces up to 99\% of vision-related attention computations and 53.9\% of FLOPs on LLaVA-v1.5 7B. It significantly outperforms existing token pruning methods and generalizes across diverse MLLMs. Beyond pruning, our insights further provide actionable guidelines for training efficient MLLMs by aligning model architecture with its intrinsic layer-wise processing dynamics. Our code is available at: https://github.com/EIT-NLP/VisiPruner.

Figures (9)

• Figure 1: Illustration of the three-stage discontinuous information processing in Multimodal Large Language Models (MLLMs). The framework separates visual-text integration into three key stages: Shallow Layers focus on task recognition, Middle Layers highlight the cross-modal fusion of sparse, task-relevant visual tokens, and Deep Layers focus on linguistic refinement after vision integration.
• Figure 2: Merge visual attention weights into a single token to stabilize the attention distribution of the first layer.
• Figure 3: Masking ranges of layers, from shallow-to-deep and deep-to-shallow, exhibit a clear reduction in cross-modal fusion at both shallow and deep layers.
• Figure 4: Top vocabulary tokens from the semantic projection of the last input token at each layer.
• Figure 5: Visualization of attention map and distribution on image with different instruction across shallow, middle and deep layers using LLaVA-v1.5 7B