ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning

TL;DR

ViTCoP tackles LVLM inefficiency by introducing a three-stage Visual-Textual Collaborative Pruning pipeline that progressively reduces visual tokens while preserving critical, diverse information. It combines a lightweight visual saliency stage, visual-textual collaboration in shallow LLM layers using VIC clustering and K-vector L2 norms, and a final textual-saliency pruning in deep LLM layers, yielding strong performance under extreme compression. Empirical results across image and video benchmarks show state-of-the-art retention, with substantial TFLOPs and memory savings (over $94\%$ TFLOPs reduction in some settings) and faster latency, indicating practical impact for deploying LVLMs in resource-constrained environments. The framework is validated on multiple LVLMs (e.g., LLaVA-1.5-7B and LLaVA-NeXT-7B/Video-7B) and demonstrates robustness through extensive ablations and efficiency analyses, highlighting the importance of staged, semantically guided pruning for multimodal models.

Abstract

Large Vision-Language Models (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the Large Language Models (LLMs). To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-Language Models demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.

Figures (6)

• Figure 1: Visual question answering results of LLaVA-1.5-7B with different pruning methods.
• Figure 2: Analysis of initial visual token redundancy. (a) A small fraction of tokens captures a majority of the attention score. (b) Model performance shows minimal degradation even when a large portion of tokens is pruned.
• Figure 3: Validation of K-vector L2 norm as a saliency proxy. (a) A strong negative correlation exists between L2 norm and attention. (b) L2 norm-based pruning is competitive with, or superior to, attention-based methods.
• Figure 4: Heatmap of visual token attention scores across LLM layers.
• Figure 5: The ViTCoP framework's three-stage process: (a) Coarse pruning in the Vision Encoder via '[CLS]' attention, (b) Collaborative pruning in shallow LLM layers using VIC clustering and K-norm merging, and (c) Aggressive text-saliency pruning in deep LLM layers.