ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models

TL;DR

ZSPAPrune tackles the high inference cost of Vision-Language Models by introducing a zero-shot, prompt-aware token pruning framework that balances task relevance and information diversity. It employs a three-stage hierarchy: prompt aggregation to a global vector $\bar{t}$, a prompt-aware core token selection based on cosine similarity, and a diversity-enrichment stage that greedily adds less-redundant tokens, yielding a final token budget $l = k + \tilde{k}$. Across multiple models (e.g., LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen2.5-VL-7B-Instruct) and benchmarks, ZSPAPrune matches or surpasses state-of-the-art pruning with up to $90\%$ token pruning and delivers substantial reductions in latency and GPU memory usage while maintaining accuracy. The approach demonstrates the importance of prompt guidance in pruning, achieving robust performance gains on several charts-dense benchmarks and offering a broadly applicable, modular method for accelerating VLM inference in zero-shot settings.

Abstract

As the capabilities of Vision-Language Models (VLMs) advance, they can process increasingly large inputs, which, unlike in LLMs, generates significant visual token redundancy and leads to prohibitive inference costs. While many methods aim to reduce these costs by pruning visual tokens, existing approaches, whether based on attention or diversity, typically neglect the guidance of the text prompt and thus fail to prioritize task relevance. In this work, we propose a novel, zero-shot method that reframes the problem by introducing a prompt-aware perspective, explicitly modeling visual token pruning as a balance between task relevance and information diversity. Our hierarchical approach first selects a core set of task-relevant visual tokens and then supplements them with diversity tokens to preserve broader context. Experiments across multiple models and benchmarks show that our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss, even when pruning up to 90\% of the tokens. Furthermore, these gains are accompanied by significant reductions in GPU memory footprint and inference latency.

Figures (3)

• Figure 1: Trade-off between task relevance and information diversity
• Figure 2: ZSPAPrune
• Figure 3: Comparison results of different $k$ (core-diversity ratio) on Qwen2.5-VL-7B-Instruct across MMMU, GQA and POPE