Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models

TL;DR

Script tackles the high visual token overhead of multimodal LLMs by marrying graph-structured pruning of redundant tokens with query-conditioned semantic pruning. By integrating a bipartite-graph redundancy estimator and a $k$-DPP-based, query-aware token selection, Script achieves large compute reductions with minimal accuracy loss, demonstrated across fourteen benchmarks and multiple MLLMs. The method is training-free, plug-and-play, and architecture-agnostic, offering practical efficiency gains for high-resolution and video inputs. Overall, Script advances efficient multimodal reasoning by explicitly balancing visual redundancy with query relevance, enabling scalable deployment on resource-constrained platforms.

Abstract

The rapid growth of visual tokens in multimodal large language models (MLLMs) leads to excessive memory consumption and inference latency, especially when handling high-resolution images and videos. Token pruning is a technique used to mitigate this issue by removing redundancy, but existing methods often ignore relevance to the user query or suffer from the limitations of attention mechanisms, reducing their adaptability and effectiveness. To address these challenges, we propose Script, a plug-and-play pruning method that requires no retraining and generalizes across diverse MLLMs. Script comprises two modules: a graph-structured pruning module that removes visually redundant tokens, and a query-conditioned semantic pruning module that preserves query-relevant visual information. Together, they enhance performance on multimodal tasks. Experiments on fourteen benchmarks across image and video understanding tasks show that Script consistently achieves higher model efficiency and predictive accuracy compared to existing pruning methods. On LLaVA-NeXT-7B, it achieves up to 6.8x prefill speedup and 10x FLOP reduction, while retaining 96.88% of the original performance.

Figures (9)

• Figure 1: (a) Efficiency Analysis on LLaVA-NeXT-7B under 88.9% reduction. (b) Comparison with other baselines on LLaVA-1.5-7B under 94.4% reduction.
• Figure 2: Token redundancy visualized via similarity and entropy on 10,000 COCO images.
• Figure 3: Overview of Script, a three-stage pruning framework: (a) overall architecture; (b) Query-Conditioned Semantic Pruning (QCSP); (c) Graph-Structured Pruning (GSP). Together, these modules remove semantically irrelevant and visually redundant tokens through a joint selection process.
• Figure 4: Visualizations of Pruning Preferences. We compute four different pruning scores for a sample from the POPE benchmark using LLaVA-1.5-7B, with the query: "Is there a person in the image?"Red indicates high preference, while blue indicates low.
• Figure B.1: Upper envelope on $\det L_S$ vs. average similarity $\rho_{\mathrm{avg}}$ for $k=5$. Unlike the trivial AM--GM bound (which is constantly $1$), the refined envelope $(1+4\rho)(1-\rho)^4$ decreases with redundancy, revealing how the maximum attainable determinant shrinks as $\rho_{\mathrm{avg}}$ grows.

Theorems & Definitions (13)

• Proposition 1: Determinant--Volume Equivalence
• proof
• Corollary 1: Hadamard Bound
• proof
• Lemma 1: Gershgorin-Based Lower Bound
• proof
• Lemma 2: AM--GM Upper Bound
• Lemma 3: Refined Upper Bound via Spectral Construction
• proof
• Proposition 2: MAP subset need not minimize redundancy