VisionZip: Longer is Better but Not Necessary in Vision Language Models

TL;DR

This work identifies substantial redundancy in visual token sequences produced by popular vision encoders and proposes VisionZip, a training-free, text-agnostic method to retain only dominant, information-rich tokens and merge the rest into contextual tokens. By aligning the reduced token set with the LLM via a brief, targeted projector fine-tuning, VisionZip delivers large efficiency gains (up to 8x faster prefilling and 2x faster inference on large models) while maintaining or surpassing state-of-the-art performance across image and video understanding tasks. The approach is broadly applicable, easy to deploy, and particularly suitable for multi-turn dialogues, making high-resolution vision-language models more practical for real-world applications. The paper also argues for a shift in focus from simply increasing token length to extracting richer visual features with lower redundancy.

Abstract

Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .

Figures (15)

• Figure 1: VisionZip Performance and Efficiency. (a) Our VisionZip significantly outperforms the current SOTA EfficientVLM model, like FastV, SparseVLM, achieving nearly 95% of the performance with only 10% of the tokens across 11 benchmarks on LLaVA-1.5. (b) VisionZip could reduce 8$\times$ prefilling time for LLaVA-NeXT 7B. (c) VisionZip reduces GPU inference time by 2$\times$ across 11 benchmarks, enabling the LLaVA-NeXT 13B model to infer faster than the 7B model while achieving better results.
• Figure 2: Redundancy Visualization. The visualization and distribution statistics of attention scores show attention concentrated on only a few tokens, while many tokens display very low attention scores, indicating significant redundancy in the visual tokens.
• Figure 3: Framework of VisionZip. VisionZip selects dominant tokens that aggregate substantial information based on visual token attention scores. Remaining tokens are merged based on semantic similarity to produce contextual tokens. VisionZip is a training-free method significantly reduces the number of image tokens, accelerating inference while maintaining performance. With efficient fine-tuning of the projector, even better results can be achieved with minimal performance loss compared to using the full token.
• Figure 4: Performance of VisionZip on the Mini-Gemini.
• Figure 5: Visualization of attention distribution across layers