D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning

TL;DR

The paper identifies two critical failure modes in MLLM token pruning: position bias in importance-based methods and structural blindness in diversity-based methods. It introduces D²Pruner, a training-free framework that combines Debiased Importance (DI) and Structural Diversity (SD), selecting a core pivot set via a debiased attention score and then applying a Maximal Independent Set–style selection on a hybrid graph to maximize both importance and spatial-semantic diversity. Across multiple models and benchmarks, D²Pruner achieves substantial FLOPs reductions with minimal accuracy loss on general understanding tasks and significantly better localization performance than prior methods, including at extreme token reductions. The approach delivers strong efficiency gains (e.g., up to 5× prefill speedup and large KV-cache reductions) and practical applicability for deploying high-resolution multimodal models in real-world settings.

Abstract

Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we find that current methods, while adequate for general understanding, catastrophically fail on fine-grained localization tasks. We attribute this failure to the inherent flaws of the two prevailing strategies: importance-based methods suffer from a strong positional bias, an inherent model artifact that distracts from semantic content, while diversity-based methods exhibit structural blindness, disregarding the user's prompt and spatial redundancy. To address this, we introduce D2Pruner, a framework that rectifies these issues by uniquely combining debiased importance with a structural pruning mechanism. Our method first secures a core set of the most critical tokens as pivots based on a debiased attention score. It then performs a Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify spatial proximity and semantic similarity. This process iteratively preserves the most important and available token while removing its neighbors, ensuring that the supplementary tokens are chosen to maximize importance and diversity. Extensive experiments demonstrate that D2Pruner has exceptional efficiency and fidelity. Applied to LLaVA-1.5-7B for general understanding tasks, it reduces FLOPs by 74.2\% while retaining 99.2\% of its original performance. Furthermore, in challenging localization benchmarks with InternVL-2.5-8B, it maintains 85.7\% performance at a 90\% token reduction rate, marking a significant advancement with up to 63. 53\% improvement over existing methods.

Figures (4)

• Figure 1: Illustration of positional bias in MLLMs and our debiasing solution.Left: Averaging attention maps over 1k images reveals a strong model bias prior (e.g., towards the bottom for LLaVA). Right: This causes the naive attention-based method (e.g., FastV) to fail by pruning salient objects like the umbrella. Our relative attention, which normalizes by this model bias prior, rectifies the pruning decision and answer correctly.
• Figure 2: Qualitative and Quantitative Analysis of D²Pruner against Prior Methods.Left: Visual comparison on localization tasks. Importance-based method, FastV, is misled by positional bias, concentrating tokens on the image bottom, while Diversity-based method, DART, suffers from structural blindness and semantic myopia. In contrast, our Debiased Importance (DI) component successfully overcomes positional bias. Our full model, D²Pruner (DI+SD), jointly addresses both semantic and spatial redundancy, resulting in a more balanced and informative token selection that enhances localization performance. Right: Quantitative evaluation of average performance on general understanding and fine-grained localization tasks. D²Pruner achieves the best performance across all reduction ratios, with notable improvements on localization tasks.
• Figure 3: Overall pipeline for our proposed D²Pruner. (a) Debiased Importance Score: The original attention map $A_{ori}$ is adjusted by subtracting the model bias prior $A_{bias}$, resulting in the relative attention map $A_{rel}$, (b) Adjacency Graph Construction: The semantic similarity and spatial adjacency are fused to create the adjacency graph $S_{adj}$. (c) Pivot-based Diverse Token Selection: In Step 1, pivots are initialized using the top-K values from $A_{rel}$, excluding neighbors as defined by $S_{adj}$; In Step 2, tokens are iteratively selected based on the highest $A_{rel}$ values, with neighbors excluded according to the adjacency graph.
• Figure 4: Ablation study.Left: Effectiveness of Debiased Importance (DI) and Structural Diversity (SD). Our full method (DI+SD) consistently outperforms FastV and the DI-only variant, especially at low token rataining ratios (e.g., 0.25, 0.1). Right: Effectiveness of spatial adjacency. Our method with $S_{spat}$ outperforms the variant without it, demonstrating the benefit of considering spatial distribution for localization tasks.