IPCV: Information-Preserving Compression for MLLM Visual Encoders

TL;DR

IPCV addresses the high computational burden of multimodal large language models by introducing a training-free, information-preserving token compression framework for Vision Transformers. It uses Neighbor-Guided Reconstruction to recover pruned tokens in later layers and Attention Stabilization to approximate pruned tokens' keys/values, followed by reintegration to preserve input token structure for the LLM. The method demonstrates superior accuracy–latency trade-offs on image and video benchmarks and generalizes across architectures, while remaining compatible with existing LLM-side pruning approaches. This work provides a practical, plug-in solution to accelerate MLLMs without sacrificing multimodal reasoning capabilities.

Abstract

Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruning strategies are inadequate: LLM-stage token pruning overlooks the ViT's overhead, while conventional ViT token pruning, without language guidance, risks discarding textually critical visual cues and introduces feature distortions amplified by the ViT's bidirectional attention. To meet these challenges, we propose IPCV, a training-free, information-preserving compression framework for MLLM visual encoders. IPCV enables aggressive token pruning inside the ViT via Neighbor-Guided Reconstruction (NGR) that temporarily reconstructs pruned tokens to participate in attention with minimal overhead, then fully restores them before passing to the LLM. Besides, we introduce Attention Stabilization (AS) to further alleviate the negative influence from token pruning by approximating the K/V of pruned tokens. It can be directly applied to previous LLM-side token pruning methods to enhance their performance. Extensive experiments show that IPCV substantially reduces end-to-end computation and outperforms state-of-the-art training-free token compression methods across diverse image and video benchmarks. Our code is available at https://github.com/Perkzi/IPCV.

Figures (6)

• Figure 1: ViT Encoding versus LLM Prefilling latency proportions across three benchmarks---MMBench (low-resolution images), OCRBench (high-resolution images), and MVBench (videos)---using Qwen2-VL-7B-Instruct and InternVL3-38B.
• Figure 2: Visualization of the delta of tokens from the shallow layers to the deep layers. (a) PCA projection of the hidden state trajectories of a pruned token and the mean of its top-10 nearest neighbors, traced from the pruning layer to the final layer of the ViT, with arrows indicating the shift direction. (b) and (c): Distributions of L1 distance and cosine similarity, computed from pairwise comparisons between the change (delta) of retained tokens and that of pruned tokens across layers. The deep-blue vertical line denotes the overall median of the pairwise distances (cosine similarities) computed between each pruned token and the mean of its top-10 most similar tokens. These visualizations reveal that tokens with high similarity tend to exhibit highly similar changes from shallow to deep layers.
• Figure 3: Overview of IPCV. IPCV prunes redundant visual tokens in the shallow layers of the vision encoder to reduce computation, then reconstructs the pruned tokens at the final layer using Neighbor-Guided Reconstruction (NGR) to deliver a semantically complete token set to the LLM. Attention Stabilization (AS) further mitigates the negative impact of token pruning by approximating the keys and values of the removed tokens in intermediate layers.
• Figure 4: Efficiency comparison of IPCV vs. ViT/LLM baselines across datasets. (a) Accuracy vs. Rel. Latency for ViT baselines in MMB. (b) LLM baselines in MMB. (c) ViT baselines in VizWiz. (d) LLM baselines in VizWiz. Rel. Latency denotes total GPU inference time normalized to Vanilla. IPCV shows consistently strong efficiency--accuracy trade-offs on both datasets.
• Figure 5: Sensitivity analysis of IPCV on MMBench-EN. (Left) Impact of pruning start layer $l_p$ and AS depth $\Delta l_{\max}$. (Right) Impact of neighbor size $k$ in NGR. ToMe is included as a reference.

Theorems & Definitions (5)

• Lemma E.3: Pointwise deviation bound
• proof
• Remark E.4
• Theorem E.5: NGR Reconstruction Error
• proof : Sketch