PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective

TL;DR

PIO-FVLM introduces a training-free, inference-objective-driven visual token reduction method for accelerating vision-language models. It leverages a layer-local proxy loss to compute gradient saliency, reorders tokens by this saliency, and applies a gradient-based NMS to select a fixed token budget, all while maintaining compatibility with FlashAttention. The approach can operate as an encoder-free method or alongside encoder compression, delivering strong results across models like LLaVA-1.5, LLaVA-NeXT, and Qwen-2.5-VL, including retaining a small fraction of tokens with minimal accuracy loss and substantial speedups and memory savings (e.g., up to $2.67\times$ prefill, $2.11\times$ total, and notable KV-cache reductions). This work demonstrates a practical, deployment-friendly path to VLM acceleration that emphasizes inference outcomes rather than reliance on attention maps or token similarity alone.

Abstract

Recently, reducing redundant visual tokens in vision-language models (VLMs) to accelerate VLM inference has emerged as a hot topic. However, most existing methods rely on heuristics constructed based on inter-visual-token similarity or cross-modal visual-text similarity, which gives rise to certain limitations in compression performance and practical deployment. In contrast, we propose PIO-FVLM from the perspective of inference objectives, which transforms visual token compression into preserving output result invariance and selects tokens primarily by their importance to this goal. Specially, vision tokens are reordered with the guidance of token-level gradient saliency generated by our designed layer-local proxy loss, a coarse constraint from the current layer to the final result. Then the most valuable vision tokens are selected following the non-maximum suppression (NMS) principle. The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment. It can be deployed independently as an encoder-free method, or combined with encoder compression approaches like VisionZip for use as an encoder-involved method. On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance, with a 2.67$\times$ prefill speedup, 2.11$\times$ inference speedup, 6.22$\times$ lower FLOPs, and 6.05$\times$ reduced KV Cache overhead. Our code is available at https://github.com/ocy1/PIO-FVLM.

Figures (5)

• Figure 1: Comparison of different token selection strategies (selecting only 12 visual tokens as an example). (a) Ours; (b) Similirty with cls token and text token based; (c) cls token similarity based and enhance diversity; (d) Cls token similarity based.More examples are given in Appendix \ref{['sec:vis_case_study_en']}.
• Figure 2: The architecture of PIO-FVLM. The overall framework consists of two stages. As shown on the left, the first stage is responsible for reordering vision tokens according to gradient saliency. The right side illustrates the second stage, which selects tokens based on the proposed NMS strategy.The $pred\_head$ is a pre-trained component native to the model, a prediction head originally designed to operate on the final-layer features.
• Figure 3: Visualization of selected important visual tokens (64 retained) using different gradient signals. Token selection is performed at Layers $[1,10,15]$: the top row (Current) ranks image tokens by gradients from the current-layer local proxy loss, while the bottom row (Last) uses gradients propagated from the final-layer output loss. Highlighted patches indicate the retained tokens.
• Figure 4: Layer-wise gradient saliency statistics. For each benchmark and layer, we report the average number of tokens per question whose gradient scores exceed the layer-wise mean. Red boxes highlight layers exhibiting notable distribution shifts for LLaVA and Qwen.
• Figure 5: Comparison of different token selection strategies . (a) Ours; (b) Similirty with cls token and text token based; (c) cls token similarity based and enhance diversity; (d) Cls token similarity based.