Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models

Abstract

Network pruning is an effective technique for enabling lightweight Large Vision-Language Models (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.

Figures (6)

• Figure 1: Illustration of the divergent statistical characteristics across different modalities, manifesting as: (a) activation representation: the textual and visual activations occupy distinct clustered regions in the representation space (t-SNE visualization); and (b) pruning importance: the pruning masks derived from the text-only and visual-only calibration data exhibit a broad IoU distribution (taking 50% sparsity level as an example)
• Figure 2: Modality decoupling via MoT probe. For each Transformer block, the QKV and FFN layers are replicated into visual and textual pathways, which process their respective token types. Independent pruning masks are derived for each pathway using activation statistics from text-only, image-only, or mixed calibration pools. This setup enables controlled comparison of modality-specific pruning sensitivity.
• Figure 3: Overview of ATV-Pruning; Color intensity reflects the degree of visual saliency. Blocks with higher visual saliency keep more salient visual tokens, with all text tokens.
• Figure 4: Ablation study about global scaling factor $\alpha$. Larger $\alpha$ allocates more salient visual tokens. A moderate range $\alpha \in [0.5, 2]$ yields stable performance across benchmarks.
• Figure 5: Effect of textual token selection under varied retained ratios during calibration, with the default visual token subset of ATV-Pruning. Removing text tokens consistently harms the performance, highlighting their necessity for stable pruning.