FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models

TL;DR

FlashVLM addresses the quadratic cost of dense visual tokens in large VLMs by introducing a text-guided, attention-light token selector that fuses extrinsic cross-modal similarity with intrinsic saliency and preserves global context through diversity-aware partitioning. It provides theoretical guarantees of sub-quadratic complexity and semantic coverage via a $\delta$-net analysis, and demonstrates beyond-lossless performance at large pruning ratios across image and video QA tasks, with architecture-agnostic applicability. The method shows strong robustness to hyperparameters and perturbations, maintains compatibility with FlashAttention, and scales effectively to high-resolution inputs and long video sequences. This yields significant efficiency gains for real-world multimodal inference while preserving or enhancing semantic grounding and generalization across diverse backbones and benchmarks.

Abstract

Large vision-language models (VLMs) typically process hundreds or thousands of visual tokens per image or video frame, incurring quadratic attention cost and substantial redundancy. Existing token reduction methods often ignore the textual query or rely on deep attention maps, whose instability under aggressive pruning leads to degraded semantic alignment. We propose FlashVLM, a text guided visual token selection framework that dynamically adapts visual inputs to the query. Instead of relying on noisy attention weights, FlashVLM computes an explicit cross modal similarity between projected image tokens and normalized text embeddings in the language model space. This extrinsic relevance is fused with intrinsic visual saliency using log domain weighting and temperature controlled sharpening. In addition, a diversity preserving partition retains a minimal yet representative set of background tokens to maintain global context. Under identical token budgets and evaluation protocols, FlashVLM achieves beyond lossless compression, slightly surpassing the unpruned baseline while pruning up to 77.8 percent of visual tokens on LLaVA 1.5, and maintaining 92.8 percent accuracy even under 94.4 percent compression. Extensive experiments on 14 image and video benchmarks demonstrate that FlashVLM delivers state of the art efficiency performance trade offs while maintaining strong robustness and generalization across mainstream VLMs.

Figures (4)

• Figure 1: VisPruner keeps its attention static and insensitive to question semantics, while FlashVLM dynamically selects key tokens based on textual cues, achieving more accurate region focus and stronger semantic alignment.
• Figure 2: The model encodes text and image features, then combines cross-modal similarity with visual attention to partition visual tokens into important, background, and diverse tokens. A semantic reweighting and redundancy elimination module selects the final key tokens from these groups, enabling efficient and semantically aligned visual reasoning.
• Figure 3: The model’s focus shifts according to each question: in the fruit image, it attends to tomatoes when asked about “the red fruit” and switches to bananas for “the yellow fruits”; in the outdoor scene, it focuses on the tree crown for leaf-related queries and moves to the blue fire hydrant when asked about its color, demonstrating strong question-dependent dynamic adaptation.
• Figure 4: The attention signal contains many noisy peaks, making key tokens hard to identify. In contrast, the text-similarity signal is sparse and semantically focused, effectively filtering out noisy attention activations. The fused score combines both, yielding more accurate and stable key-token localization.

Theorems & Definitions (17)

• Remark 6.2
• Remark 6.4
• Theorem 6.5: Amortized Computational Efficiency
• proof
• Definition 6.6: $\delta$-Cover
• Lemma 6.7: Local Coverage Condition
• proof
• Theorem 6.8: Global $\delta$-Net Guarantee
• proof
• Remark 6.9: Implication vs. Top-K