IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning

TL;DR

This paper reveals that LVLMs implicitly establish visual coordinates through Rotary Position Embeddings, identifying specific token positions as implicit coordinates (IVC) critical for spatial reasoning. Building on this insight, it introduces IVC-Prune, a training-free, prompt-aware pruning strategy that preserves both IVC tokens and semantically relevant foreground tokens, using a two-stage foreground selection and a single, layer-wise pruning decision to maximize KV-cache reduction. Across four LVLMs and twenty benchmarks, IVC-Prune reduces visual tokens by ~50% while maintaining ≥99% of original performance and often improving grounding and VQA results, with notable efficiency gains in decoding. The method’s robustness across architectures and tasks, along with its theoretical link between RoPE and spatial reasoning, offers a practical path to efficient high-resolution multiview understanding and invites future work on dynamic pruning and extensions to other positional encodings.

Abstract

Large Vision-Language Models (LVLMs) achieve impressive performance across multiple tasks. A significant challenge, however, is their prohibitive inference cost when processing high-resolution visual inputs. While visual token pruning has emerged as a promising solution, existing methods that primarily focus on semantic relevance often discard tokens that are crucial for spatial reasoning. We address this gap through a novel insight into \emph{how LVLMs process spatial reasoning}. Specifically, we reveal that LVLMs implicitly establish visual coordinate systems through Rotary Position Embeddings (RoPE), where specific token positions serve as \textbf{implicit visual coordinates} (IVC tokens) that are essential for spatial reasoning. Based on this insight, we propose \textbf{IVC-Prune}, a training-free, prompt-aware pruning strategy that retains both IVC tokens and semantically relevant foreground tokens. IVC tokens are identified by theoretically analyzing the mathematical properties of RoPE, targeting positions at which its rotation matrices approximate identity matrix or the $90^\circ$ rotation matrix. Foreground tokens are identified through a robust two-stage process: semantic seed discovery followed by contextual refinement via value-vector similarity. Extensive evaluations across four representative LVLMs and twenty diverse benchmarks show that IVC-Prune reduces visual tokens by approximately 50\% while maintaining $\geq$ 99\% of the original performance and even achieving improvements on several benchmarks. Source codes are available at https://github.com/FireRedTeam/IVC-Prune.

Figures (4)

• Figure 1: Implicit Visual Coordinate (IVC) tokens are crucial for spatial reasoning in LVLMs. Left: Visual grounding examples under different input settings. Top right: RoPE cosine and sine components across token positions, with IVC token locations (10% of total) marked in black. Bottom right: RefCOCO accuracy across three LVLMs under varying input settings, showing that adding IVC tokens largely restores performance. Detailed results and analysis are provided in Appendix \ref{['sec:ex_fig1']}.
• Figure 2: Illustration of the IVC-Prune strategy. Token selection is performed once at layer $i$ on visual tokens, while preserving their original position IDs. The selection decision prunes the corresponding tokens from the KV caches of all earlier layers and is used for subsequent layers.
• Figure 3: Comparison of image preprocessing strategies in different LVLMs. White squares represent image tokens. Red squares indicate special tokens introduced in DeepSeek-VL2.
• Figure 4: Visualization of the positional embedding scores for four LVLMs, where cosine ($V(m)$) and sine ($U(m)$) components are summed over all dimensions as in Eqs. \ref{['eq:V_m']} and \ref{['eq:U_m']}. Black squares denote the selected 10% IVC tokens, and red squares indicate the special tokens introduced in DeepSeek-VL2. Note that IVC tokens are determined solely by position and are independent of the content.