Towards Lossless Ultimate Vision Token Compression for VLMs

TL;DR

This work tackles the inefficiency of visual language models caused by redundant visual tokens. It introduces LUVC, a training free framework with Orthogonal Iterative Merger for 2D token merging in the visual encoder and Spectrum Pruning Unit for cascade frequency based pruning in the LLM, achieving roughly 2x speedup with minimal accuracy loss. The approach demonstrates strong generalization across video, single image, multi image, and dense chart document tasks, while maintaining compatibility with FlashAttention. LUVC is validated across multiple VLM architectures and datasets, showing notable improvements over existing methods with modest or negligible performance degradation. The practical impact is substantial, enabling faster deployment of high resolution VLMs in real time applications without retraining.

Abstract

Visual language models encounter challenges in computational efficiency and latency, primarily due to the substantial redundancy in the token representations of high-resolution images and videos. Current attention/similarity-based compression algorithms suffer from either position bias or class imbalance, leading to significant accuracy degradation. They also fail to generalize to shallow LLM layers, which exhibit weaker cross-modal interactions. To address this, we extend token compression to the visual encoder through an effective iterative merging scheme that is orthogonal in spatial axes to accelerate the computation across the entire VLM. Furthermoer, we integrate a spectrum pruning unit into LLM through an attention/similarity-free low-pass filter, which gradually prunes redundant visual tokens and is fully compatible to modern FlashAttention. On this basis, we propose Lossless Ultimate Vision tokens Compression (LUVC) framework. LUVC systematically compresses visual tokens until complete elimination at the final layer of LLM, so that the high-dimensional visual features are gradually fused into the multimodal queries. The experiments show that LUVC achieves a 2 speedup inference in language model with negligible accuracy degradation, and the training-free characteristic enables immediate deployment across multiple VLMs.

Figures (7)

• Figure 1: The distribution of image-text attention scores and tokens spectrum. Due to position bias, the attention scores exhibit a significant upward trend near the text region, whereas the spectrum curve remains position-invariant.
• Figure 2: Visualization of low-frequency visual tokens of Internvl2.5-8B in layer-8.
• Figure 3: The overview of LUVC. LUVC consists of two key components. (1) Orthogonal Iterative Merger (OIM), which maintains the spatial structure while increasing computational parallelism by performing step-by-step merging in both lateral and longitudinal dimensions. (2) Spectrum Pruning Unit (SPU), which applies low-pass filtering to visual tokens through FFT and IFFT, which achieves progressive token pruning via cascade structure.
• Figure 4: The examples of 2D projectors demonstrate that they rely on the 2D spatial structure.
• Figure 5: Performance comparison of LUVC and PACT under different Speedup.