Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference

TL;DR

Efficient Video Sampling (EVS) tackles the token-count explosion in video-enabled vision-language models by pruning temporally static patches, reducing the input token burden without architectural changes or retraining. It offers two pruning pathways—RGB-space and embedding-space—and supports position-preserving versus sequential token IDs, enabling flexible deployment. A light uptraining phase with stochastic pruning rates further improves robustness across compression levels and preserves performance across a range of $q$ at inference. EVS yields substantial TTFT and KV-cache reductions, enabling longer video contexts and faster inference while maintaining semantic fidelity, making scalable video-language understanding more practical for real-world applications.

Abstract

Vision-language models (VLMs) have recently expanded from static image understanding to video reasoning, but their scalability is fundamentally limited by the quadratic cost of processing dense frame sequences. Long videos often exceed the token budget of modern language models, leading to severe context limitations and latency issues. We introduce Efficient Video Sampling (EVS), a simple, plug-and-play method for reducing token redundancy in videos by identifying and pruning temporally static patches -- spatial regions that remain unchanged across consecutive frames. EVS preserves positional identity, requires no architectural changes or retraining. We show that EVS substantially reduces token count while maintaining semantic fidelity, enabling faster inference and longer input sequences. Applied at inference time, EVS reduces large language model (LLM) time-to-first-token (TTFT) by up to 4x with minimal accuracy loss. When combined with an uptraining phase using stochastic pruning rates, EVS yields models that are robust to varying compression levels and retain full performance under aggressive pruning. Extensive experiments demonstrate that EVS consistently improves efficiency-accuracy trade-offs, unlocking scalable video-language understanding without sacrificing quality.

Figures (5)

• Figure 1: Efficient Video Sampling: Pruning Static Patches. This example video illustrates the key idea behind our Efficient Video Sampling (EVS) method. The top row displays a sequence of original video frames that depict a dog playing with a bone. The second and third rows shows the same sequence after EVS with a sequence-level pruning ratio of $q=50\%$ and $q=75\%$ respectively. Importantly, EVS does not prune each frame uniformly: more dynamic frames are pruned less aggressively, while static frames are pruned more heavily. EVS selectively retains only the most dynamic and informative patches—such as the dog's head playing with the bone. This enables significant token reduction without sacrificing semantic content, making EVS particularly effective for videos with sparse motion.
• Figure 2: Accuracy vs. percentage of tokens dropped by EVS. This figure illustrates the trade-off between inference acceleration (measured as the percentage of tokens dropped, which directly affects TTFT) and the accuracy score, between two approaches: uptraining the model (solid lines) versus using EVS as a plug-and-play post-processing step without uptraining (dashed lines). Each color indicates a dataset. Uptraining consistently mitigates performance drops, especially at higher pruning rates. Notably, one can choose a pruning factor $q$ to achieve the desired tradeoff between TTFT reduction and accuracy. See Appendix \ref{['app:additional_results']} for additional details on the impact of pruning rates on the accuracy score.
• Figure 3: Visual explanation of Sequential and Position-preserving handling of pruned tokens. The top row indicates position IDs assigned to input images and text prompts without pruning. Empty cells correspond to pruned tokens. In Sequential method, position ids for input tokens are computed for output tokens after pruning. In Position-preserving method, position IDs are computed for token positions before pruning.
• Figure 4: Measured TTFT Speedup for different pruning rates. We measure and report TTFT speedup for the LLM backbone (a), as our method mainly focuses on reducing the number of input tokens into the LLM backbone. We report the overall VLM speedup (b) for completeness.
• Figure 5: Calculated LLM KV-cache size reduction for different pruning rates.