FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging

TL;DR

FlashVID tackles the computational bottleneck of Video Large Language Models by compressing visual tokens in a training-free, plug-and-play manner. It combines Attention and Diversity-based Token Selection (ADTS) to pick semantically informative and diverse tokens per frame, with Event relevance calibration and [CLS] attention to guide selection, and Tree-based Spatiotemporal Token Merging (TSTM) to form spatiotemporal redundancy trees for joint reduction across frames. Across five benchmarks and three VLLMs, FlashVID achieves up to 99.1% of full-token accuracy at only 10% retention and enables processing up to 10x more frames under the same budget, yielding substantial efficiency gains (e.g., 6.3x prefilling, 2.1x TTFT). This demonstrates that training-free, structure-aware spatiotemporal compression can extend temporal context and maintain performance, broadening the accessibility of VLLMs in resource-constrained settings.

Abstract

Although Video Large Language Models (VLLMs) have shown remarkable capabilities in video understanding, they are required to process high volumes of visual tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes Attention and Diversity-based Token Selection (ADTS) to select the most representative tokens for basic video representation, then applies Tree-based Spatiotemporal Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five video understanding benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at https://github.com/Fanziyang-v/FlashVID.

Figures (10)

• Figure 1: Performance of FlashVID. (a) TTM may merge less correlated visual tokens, failing to capture fine-grained video dynamics. (b) FlashVID can enable Qwen2.5-VL to process $\textbf{10}\times$ video frames, significantly improving the relative performance by 8.6% while maintaining overall computational budget. (c) FlashVID significantly outperforms current SOTA acceleration frameworks (e.g., FastV, VisionZip, FastVID) on three representative VLLMs.
• Figure 2: Efficient inference paradigms. State-of-the-art acceleration frameworks can be mainly divided into three categories: 1) Before-LLM Compression; 2) Inner-LLM Pruning; and 3) Hybrid Compression, where the hybrid compression can be viewed as a trade-off of the Before-LLM Compression and Inner-LLM Pruning strategy.
• Figure 3: Number of merged tokens per frame with TSTM (orange) and TTM (blue) under the same threshold, with average merging similarity differences between TSTM and TTM shown in green.
• Figure 5: Overview of our FlashVID. FlashVID compresses visual tokens by two synergistic modules: (1) ADTS prioritizes spatiotemporally informative tokens while ensuring feature diversity by solving a calibrated Max-Min Diversity Problem (MMDP); (2) TSTM models redundancy by spatiotemporal redundancy trees, which effectively capture fine-grained video dynamics.
• Figure 6: Visualizations of Tree-based Spatiotemporal Token Merging (TSTM). We select three consecutive video frames that show obvious variations in spatial locations, scale, and orientation for each case to illustrate the advantages of our TSTM in FlashVID. TSTM jointly models spatial and temporal redundancy via spatiotemporal redundancy trees for capturing fine-grained spatiotemporal relationships; thus, it achieves better spatiotemporal redundancy compression.