Object-Centric Vision Token Pruning for Vision Language Models

TL;DR

The paper tackles the heavy computation cost of dense vision tokens in vision-language models by introducing OC-VTP, a guaranteed token pruning framework. It builds a lightweight OC-pruner based on Slot Attention to select the most representative tokens via a reconstruction objective, trained on COCO and inserted between the vision encoder and projection to VLMs without downstream fine-tuning. The approach delivers strong accuracy retention across budgets, reduces FLOPs by up to an order of magnitude, and provides interpretable, object-centric token selections that generalize well to unseen data. The work demonstrates practical impact by enabling efficient VLM inference on multiple architectures (e.g., LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL) with minimal overhead and broad applicability to real-world multimodal tasks.

Abstract

In Vision Language Models (VLMs), vision tokens are quantity-heavy yet information-dispersed compared with language tokens, thus consume too much unnecessary computation. Pruning redundant vision tokens for high VLM inference efficiency has been continuously studied but all existing methods resort to indirect and non-guaranteed ways. We propose OC-VTP, a direct and guaranteed approach to select the most representative vision tokens for high-efficiency yet accuracy-preserving VLM inference. Our OC-VTP requires merely light-weight pre-training of a small object-centric vision token pruner, which can then be inserted into existing VLMs, without fine-tuning of any models on any datasets. It is gauranteed that the most representative vision tokens are kept by minimizing the error in reconstructing the original unpruned tokens from the selected ones. Across any vision pruning ratios, i.e., inference efficiency, our OC-VTP consistently helps mainstream VLMs to preserve the highest inference accuracy. Our pruning also demonstrates interesting interpretability. Our codes are available at https://github.com/GarryLarry010131/OC-VTP.

Figures (4)

• Figure 1: (left) Our OC-VTP consistently outperforms prior SotA methods, retaining over 95% of accuracy with only 11.1% of visual tokens on LLaVA-1.5. (right) Our OC-VTP reduces FLOPs by nearly 85% on LLaVA-1.5-7B when retaining 11.1% vision tokens, and by 95% on LLaVA-NeXT-7B when retaining 5.6% vision tokens, at a text length of 32, assuming MAC=2.
• Figure 2: (left) Structure of typical Vision-Language Models (VLMs), and three Vision Token Pruning (VTP) places: encoder methods like ToMe and FastV ToMeFastV; decoder methods like SparseVLM and PyramidDrop SparseVLMPDrop; and in-between methods like TRIM, VisionZip and HiPrune TRIMVisionZiphiprune. Our OC-VTP operates in-between. (right) OC-pruner is the core of our OC-VTP. It takes the middle layer tokens from the vision encoder as input, and employs queries sampled from a learned Gaussian distribution to locate the most representative token of each object or object part. The corresponding indexes are used to prune the vision tokens.
• Figure 3: Inference time per image (ms). Average inference time for each case/image is calculated from total evaluation using LMMs-Eval on a single V100-32G GPU with $\text{batch size}=1$. Notably, Random is a random pruning baseline with no extra cost.
• Figure 4: OC-VTP visualization results. OC-Pruner retains one token each slot, and the slots represent different objects in the scene. The retaining budget is 64, and the test is conducted on LLaVA-1.5.