AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance

TL;DR

AdaptInfer tackles the high inference cost of Vision-Language Models caused by dense vision-token processing. By exploiting dynamic text guidance derived from layer-wise text-to-text attention and cross-attention shift analysis, it reweights text-to-vision attention to prune vision tokens adaptively at strategically chosen layers. The method is training-free, plug-and-play, and transferable across multiple VLMs and tasks, delivering substantial latency reductions (e.g., ~61% CUDA latency improvement) while preserving high accuracy (e.g., ~93% average at 64 tokens on LLaVA-1.5-7B). Across diverse benchmarks, AdaptInfer consistently outperforms static prompt-based or heuristic pruning baselines, demonstrating both effectiveness and practical impact for real-time multimodal reasoning.

Abstract

Vision-language models (VLMs) have achieved impressive performance on multimodal reasoning tasks such as visual question answering, image captioning and so on, but their inference cost remains a significant challenge due to the large number of vision tokens processed during the prefill stage. Existing pruning methods often rely on directly using the attention patterns or static text prompt guidance, failing to exploit the dynamic internal signals generated during inference. To address these issues, we propose AdaptInfer, a plug-and-play framework for adaptive vision token pruning in VLMs. First, we introduce a fine-grained, dynamic text-guided pruning mechanism that reuses layer-wise text-to-text attention maps to construct soft priors over text-token importance, allowing more informed scoring of vision tokens at each stage. Second, we perform an offline analysis of cross-modal attention shifts and identify consistent inflection locations in inference, which inspire us to propose a more principled and efficient pruning schedule. Our method is lightweight and plug-and-play, also generalizable across multi-modal tasks. Experimental results have verified the effectiveness of the proposed method. For example, it reduces CUDA latency by 61.3% while maintaining an average accuracy of 93.1% on vanilla LLaVA-1.5-7B. Under the same token budget, AdaptInfer surpasses SOTA in accuracy.

Figures (7)

• Figure 1: Preliminary observations. (a) The consistent low mIoU shows the key text token varies across layers. (b)(c) Layer-wise distribution of attention shifts on MME and TextVQA, which shows a highly consistent trend.
• Figure 2: The architecture of AdaptInfer. Text token importance is computed and guides vision token selection at every pruning layer adaptively. The right panel illustrates the internal computation details of the Rank & Prune module. Step I: Compute the text token prior. Step II: Use the transposed prior to reweight the t2v attention. Step III: Rank the vision tokens and prune the least informative ones.
• Figure 3: Performance trends on two datasets. AdaptInfer outperforms SparseVLM and PDrop across all retained token budgets.
• Figure 4: Layer-wise distribution of attention shifts on LLava-1.5-13B.
• Figure 5: Layer-wise distribution of attention shifts on Qwen2-VL-2B on MME.