Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering

TL;DR

This work tackles visual token pruning in multimodal LLMs by modeling the trade-off between visual preservation and prompt alignment through a closed-form error bound based on the Hausdorff distance $d_H$, and an $\\ extepsilon$-covering framework to quantify prompt-visual coupling. It introduces Multi-Objective Balanced Covering (MoB), a training-free pruning method that recasts pruning as bi-objective covering with budgets $K_p$ and $K-K_p$, solved via greedy radius trading using a two-stage process (k-fold NN covering for prompts and FPS for visuals) and backed by provable guarantees with complexity $T_{MoB}=\mathcal{O}(N(L+K)d)$. Empirically, MoB preserves up to $96.4\%$ of performance with only $11.1\%$ of visual tokens on LLaVA-1.5-7B and accelerates LLaVA-Next-7B by $1.3$–$1.5\times$, with strong generalization to Qwen2-VL and Video-LLaVA across 14 benchmarks. The approach offers a scalable, training-free framework for adaptive pruning in advanced MLLMs, with potential applicability to other redundancy-heavy domains.

Abstract

Existing visual token pruning methods target prompt alignment and visual preservation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging $ε$-covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Covering (MoB), which reformulates visual token pruning as a bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers a provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation to challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4% of performance for LLaVA-1.5-7B using only 11.1% of the original visual tokens and accelerates LLaVA-Next-7B by 1.3-1.5$\times$ with negligible performance loss. Additionally, evaluations on Qwen2-VL and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-language tasks.

Figures (9)

• Figure 1: (a) Comparison of single- vs. bi-objective pruning methods on LLaVA-1.5-7B at a $66.7\%$ pruning rate; (b) distribution of the prompt-visual coupling, revealing two distinct patterns across various tasks: weak coupling (large distance) and strong coupling (small distance); (c) radar charts of LLaVA-1.5-7B with visual tokens reduced from $576$ to $192$, $128$, and $64$ (left-to-right), demonstrating the consistent improvements of MoB across 10 well-recognized benchmarks.
• Figure 2: Illustration of prompt-visual coupling with two distinct patterns: In fine-grained tasks (e.g. POPE), only a few patches are critical, so the worst-case patch lies far from best-case ones, resulting in a large Hausdorff distance and making prompt alignment valuable. In coarse-grained tasks (e.g. MMB), many relevant patches contain the answer cues; thus, the worst-case patch remains close to best-case ones, yielding a small Hausdorff distance and making visual preservation more efficient.
• Figure 3: Performance-Latency trade-off comparisons across four benchmarks on LLaVA-Next-7B.
• Figure 4: Comprehensive ablation on the budget configuration $\langle K_{\rm p}, K\rangle$ across four benchmarks with distinct prompt-visual coupling $\eta$ on LLaVA-1.5-7B, where $K=\{64, 128, 192\}$; the mean relative slope (%) is given by $\tfrac{100}{x_n-x_1}\sum_{i=1}^{n-1}\tfrac{y_{i+1}-y_i}{y_i}$, quantifying the trade-off intensity; the ratio $\tfrac{K_{\rm p}}{K}$ reflects the cost-effectiveness of prompt alignment, and the box plot presents the distribution of $\eta$.
• Figure 5: Ablation on the ratio of $k/K_{\rm p}$.

Theorems & Definitions (24)

• Lemma 1: An Error Bound for Visual Token Pruning
• Remark
• Lemma 2: A Relaxed Error Bound under Practical Budgets
• Definition 1: $\epsilon$-cover, Covering Number, and Covering Regularity
• Lemma 3: Covering Number Bounds
• Remark
• Theorem 1: Trade-off between Prompt Alignment and Visual Preservation
• Remark : Optimal Attainment Level
• Remark : Effect of Budget and Coupling Strength
• Theorem 2: Performance Guarantee