HoliTom: Holistic Token Merging for Fast Video Large Language Models

TL;DR

HoliTom tackles the token explosion in video LLMs by coupling a training-free holistic pruning pipeline that first applies global temporal segmentation (outer-LLM) and spatial merging with a robust inner-LLM token similarity merging. The temporal step uses dynamic programming to maximize prune-able features via a global redundancy metric; spatial merging employs attention-based token selection and density-peak clustering to handle residual redundancy, while inner-LLM merging averages selected tokens to preserve information. Empirical results on LLaVA-OneVision-7B and LLaVA-Video-7B show major efficiency gains (FLOPs down to 6.9% of the original) with negligible accuracy loss (approx. 99.1%), along with faster Time-To-First-Token and higher decoding throughput, validating the benefits of integrating outer- and inner-LLM pruning for long-form video understanding. The work provides a practical framework for deploying efficient video LLMs by intelligently distilling input tokens without costly model fine-tuning.

Abstract

Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

Figures (14)

• Figure 1: Left: We introduce HoliTom, a training-free holistic token merge method for fast video LLMs. Its key innovation lies in its global, redundancy-aware outer-LLM spatio-temporal compression and robust, token similarity-based inner-LLM compression. Right: The Efficiency/Performance trade-off curve of multiple training-free methods on four widely used video understanding benchmarks: MVBench, EgoSchema, LongVideoBench, and VideoMME. Our method, HoliTom, surpasses the SoTA approaches by maintaining 99.1% average performance while reducing FLOPs to 6.9%.
• Figure 2: Overview of our HoliTom method.HoliTom compresses video LLMs across three scopes; the first two are outer-LLM pruning. Temporal Merging maximizes temporal compression via global redundancy-aware segmentation, merging similar tokens into their first occurrence. Spatial Merging further reduces redundancy by applying tailored spatial compression based on the characteristics of remaining temporal variations. Inner-LLM Merging leverages attention within the LLM to identify key tokens and merges less important, similar tokens, streamlining information within the LLM.
• Figure 3: Left: Performance of our method vs. FastV when pruning various layers at rate R=50%. Right: Performance comparison with varying pruning rates at a fixed layer (K=14).
• Figure 4: Performance vs. number of frames for our method and other token compression methods.
• Figure 5: Achieving superior inference. "Other" indicates token pre-processing time (e.g., pooling). Our proposed method reduces Time-To-First-Token (TTFT) by 2.28$\times$ and achieves 1.32$\times$ higher decoding throughput, outperforming all other token compression methods and the vanilla model.