Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models

TL;DR

This work tackles the inefficiency of VideoLLMs due to heavy visual token loads by diagnosing design and implementation limitations of existing token compression methods. It introduces VidCom2, a two-stage, plug-and-play framework that adaptively allocates frame-wise token budgets based on frame uniqueness and preserves distinctive tokens through an adaptive, within-frame and cross-video token selection. Across diverse benchmarks and VideoLLMs, VidCom2 achieves near-original performance with substantially reduced latency (e.g., 25% visual tokens yielding 99.6% accuracy and ~70.8% LLM latency reduction), while remaining compatible with efficient attention operators. The results demonstrate strong efficiency-accuracy trade-offs and highlight the framework's broad applicability to other token compression methods.

Abstract

Video large language models (VideoLLM) excel at video understanding, but face efficiency challenges due to the quadratic complexity of abundant visual tokens. Our systematic analysis of token compression methods for VideoLLMs reveals two critical issues: (i) overlooking distinctive visual signals across frames, leading to information loss; (ii) suffering from implementation constraints, causing incompatibility with modern architectures or efficient operators. To address these challenges, we distill three design principles for VideoLLM token compression and propose a plug-and-play inference acceleration framework "Video Compression Commander" (VidCom2). By quantifying each frame's uniqueness, VidCom2 adaptively adjusts compression intensity across frames, effectively preserving essential information while reducing redundancy in video sequences. Extensive experiments across various VideoLLMs and benchmarks demonstrate the superior performance and efficiency of our VidCom2. With only 25% visual tokens, VidCom2 achieves 99.6% of the original performance on LLaVA-OV while reducing 70.8% of the LLM generation latency. Notably, our Frame Compression Adjustment strategy is compatible with other token compression methods to further improve their performance. Our code is available at https://github.com/xuyang-liu16/VidCom2.

Figures (6)

• Figure 1: Power of frame uniqueness. Removing 24 redundant frames results in accurate video understanding by VideoLLMs, while dropping just 8 unique frames leads to inaccurate video comprehension, highlighting the critical role of unique frames for VideoLLMs.
• Figure 2: Overall framework of VidCom$^2$. Our VidCom$^2$ performs plug-and-play token compression in two stages: (i) Frame Compression Adjustment: adjusts compression intensity based on frame uniqueness (see Figure \ref{['fig:frame_uniqueness_vidcom2']}), (ii) Adaptive Token Compression: preserves tokens based on their within-frame and cross-video uniqueness.
• Figure 3: Visualization of frame uniqueness quantified by our VidCom$^2$. Taller and darker bars indicate frame uniqueness, where VidCom$^2$ allocates more tokens to unique frames to preserve critical visual information.
• Figure 4: Performance with Qwen2-VL. At $R=25\%$, VidCom$^2$ surpasses DyCoke and SparseVLM by 7.6% and 4.6% of original performance in long video tasks.
• Figure 5: Effects of Frame Compression Adjustment on other methods."+VidCom$^2$" indicates the application of our Frame Compression Adjustment strategy.