LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models

TL;DR

LVPruning introduces language-guided vision token pruning for multi-modal LLMs by embedding lightweight cross-attention decision modules that assign keep/remove relevance scores to vision tokens. Training freezes the base model and optimizes only the pruning modules with a causal language modeling objective plus a ratio-consistency loss, while inference selectively retains tokens per layer based on learned scores. The approach yields substantial efficiency gains, achieving up to $62.1\%$ TFLOPs reduction and pruning up to $90\%$ of vision tokens by the middle layers, with an average accuracy loss around $0.45\%$ across nine multimodal benchmarks. LVPruning demonstrates competitive or superior performance relative to state-of-the-art Q-former–based models while maintaining favorable compute-accuracy tradeoffs, highlighting practical applicability for resource-constrained MLLMs.

Abstract

Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.

Figures (6)

• Figure 1: LVPruning can reduce 62.1% of inference TFLOPs for LLaVA-1.5-7B with marginal performance loss across nine multi-modal benchmarks. *All TFLOPs reported in this paper are computed using a dummy input consisting of 1 image and 30 text tokens.
• Figure 2: Overall Framework Architecture. LVPruning modules are incorporated into specific layers of an MLLM, where vision tokens serve as queries and language tokens act as keys and values. A pruning decision is predicted for each vision token. The operation denoted by $\otimes$ applies these decisions—serving as attention masking during training and token removal via indexing during inference.
• Figure 3: Performance variance between LLaVA-1.5-7B DBLP:journals/corr/abs-2310-03744 and LVPruning with different vision token kept ratios $\rho$ on nine multi-modal benchmarks. Even at a low token kept ratio, such as $\rho=0.45$, the performance degradation remains small.
• Figure 4: Comparison of Inference FLOPs of LLaVA-1.5-7B DBLP:journals/corr/abs-2310-03744 and LVPruning with different vision token kept ratio $\rho$.
• Figure 5: The relationship between inference TFLOPs and the performance of LVPruning and other state-of-the-art MLLMs evaluated on the GQA benchmark DBLP:conf/cvpr/HudsonM19. LVPruning (green area) outperforms Q-former-based models (blue area) while achieving substantial computational savings compared to LLaVA-1.5 (red area) DBLP:journals/corr/abs-2310-03744. This demonstrates LVPruning’s ability to balance performance and efficiency among state-of-the-art MLLMs.