Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models

TL;DR

AdaptPrune tackles the high computational cost of large vision-language models by introducing a training-free, multi-cue visual token pruning method. By reframing pruning as an adaptive NMS problem that fuses attention, spatial distance, and token similarity, it achieves robust performance even at aggressive pruning ratios (up to 90%). The approach includes distribution correction to mitigate positional bias and an iterative suppression mechanism with defined decay terms, resulting in improved token diversity and preserved semantic content across diverse LVLMs and benchmarks. The findings demonstrate meaningful speedups and memory savings with minimal accuracy loss, offering practical impact for real-world multimodal systems.

Abstract

As the computational needs of Large Vision-Language Models (LVLMs) increase, visual token pruning has proven effective in improving inference speed and memory efficiency. Traditional pruning methods in LVLMs predominantly focus on attention scores to determine token relevance, overlooking critical aspects such as spatial position and token similarity. To this end, we introduce AdaptPrune, a novel plug-and-play training-free pruning method that builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach. Our method is based on several observed phenomena in large models: the positional bias in the model's image attention and the redundancy of token information ignored by previous approaches. By integrating attention, spatial, and similarity information, our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions. Our method has been extensively tested across various LVLMs and benchmarks, confirming its robustness and adaptability. The results demonstrate that AdaptPrune consistently outperforms existing methods across various pruning ratios. Code is available at https://github.com/bzluan/AdaptPrune.

Figures (9)

• Figure 1: Comparison between FastV fastv and our AdaptPrune at different pruning ratios. FastV fastv utilizes only attention scores for token pruning, while AdaptPrune incorporates attention, spatial, and similarity information for a more holistic investigation.
• Figure 2: Visualization of attention positional bias phenomenon. The average attention distribution projected onto image patches exhibits a fixed pattern across different layers and tasks.
• Figure 3: Visualization of the information redundancy phenomenon. The tokens with the highest attention score are kept, most of which are adjacent and have high similarity.
• Figure 4: An overview of our AdaptPrune framework. AdaptPrune employs a simple and effective one-layer pruning strategy and incorporates an adaptive NMS algorithm to optimize the pruning process. AdaptPrune dynamically adjusts the suppression process using three critical cues: attention score, spatial distance, and token similarity.
• Figure 5: Performance comparison of FastV fastv and AdaptPrune on the LLaVA-1.5-7B model at different pruning ratios across eight evaluation benchmarks. The green line represents the LLaVA-1.5-7B baseline results without pruning.