CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models

TL;DR

This work tackles the high inference cost of large vision-language models by revealing that attention scores alone poorly reflect token importance and that vision-token FFNs exhibit redundancy in intermediate layers. It introduces CAPA, a dual strategy combining Contribution-Aware Pruning (using Attention Contribution $C_i$ to select informative visual tokens) and FFN Approximation (replacing redundant FFNs with a learned Hadamard product $\mathbf{y}^{(l)} \approx \mathbf{x}^{(l)} \odot \boldsymbol{\alpha}$, solved in closed form). The approach identifies a heterogeneous class of attention sinks (Probability Dumps vs Structural Anchors) and layer-wise FFN redundancy, enabling a 78% FLOPs reduction with minimal accuracy loss across multiple LVLM backbones and benchmarks. Experimental results show CAPA outperforms attention-based pruning baselines, particularly at phase-transition layers, and maintains robustness on tasks requiring fine-grained perception and reasoning. The work paves the way for efficient LVLM inference through targeted token retention and computation-aware FFN designs.

Abstract

Efficient inference in Large Vision-Language Models is constrained by the high cost of processing thousands of visual tokens, yet it remains unclear which tokens and computations can be safely removed. While attention scores are commonly used to estimate visual token importance, they are an imperfect proxy for actual contribution. We show that Attention Contribution, which weights attention probabilities by value vector magnitude, provides a more accurate criterion for visual token selection. Our empirical analysis reveals that visual attention sinks are functionally heterogeneous, comprising Probability Dumps with low contribution that can be safely pruned, and Structural Anchors with high contribution essential for maintaining model performance. Further, we identify substantial redundancy in Feed-Forward Networks (FFNs) associated with visual tokens, particularly in intermediate layers where image tokens exhibit linear behavior. Based on our findings, we introduce CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework that prunes visual tokens using attention contribution at critical functional transitions and reduces FFN computation through efficient linear approximations. Experiments on various benchmarks across baselines show that CAPA achieves competent efficiency--performance trade-offs with improved robustness.

Figures (9)

• Figure 1: Overall framework of CAPA — (Left) Contribution-Aware Pruning: at each generation step $t$, we compute the Attention Contribution score $C_i$ for every visual token by weighting its value vector magnitude with the attention probability assigned by the last generated token query $q_t$; (Right) FFN Approximation: in the layers identified as redundant (high input-output cosine similarity), we replace the computationally expensive dense FFNs ($\mathcal{O}(d^2)$) with a lightweight, learned element-wise Hadamard product ($\mathcal{O}(d)$).
• Figure 2: Sink distribution analysis across LLaVA-1.5 and Qwen2.5-VL. Using a sink threshold of $\tau=20$, we classify tokens into two groups: low-contribution Probability Dumps (Type I) and high-contribution Structural Anchors (Type II) . The dashed line represents the sink identification threshold.
• Figure 3: Layer-wise FFN Redundancy for Image vs. Text Tokens. The plot displays the mean cosine sim. $\cos(\mathbf{x}, \mathbf{x} + \mathrm{FFN}(\mathbf{x}))$ across network layers, evaluated on 500 samples from the MSCOCO 2017 val set.
• Figure 4: Comparative analysis of FFN redundancy across LVLM architectures. We plot the average cosine sim. between the input hidden state $\mathbf{x}$ and the residual output $\mathbf{x} + \text{FFN}(\mathbf{x})$ for three baselines.
• Figure 5: Hellinger distance between the vanilla model and pruned logits across generation steps, comparing attention-score and attention-contribution pruning for 3 baselines on the COCO2017 val. set (200 examples).