KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs

TL;DR

Training-free video understanding is hindered by temporal redundancy and large visual token budgets. KTV introduces a two-stage framework that first clusters frame features to select diverse keyframes and then prunes per-frame visual tokens using a balance of importance, redundancy, and question relevance, enabling concise, informative inputs to a pre-trained VLM. Across seven MC VideoQA benchmarks, KTV achieves state-of-the-art or competitive performance while using far fewer tokens and offering faster inference, demonstrating strong efficiency with minimal or no video-specific training. This approach offers practical benefits for long videos and resource-constrained deployments, and provides a solid foundation for future training-based enhancements.

Abstract

Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.

Figures (6)

• Figure 1: The pipeline of our two-stage training-free Keyframes and Key Tokens selection for empowering LLMs in Video understanding, termed KTV. Given a video clip ($1^{st}$ row), KTV first performs question-agnostic keyframe selection ($3^{rd}$ row), identifying representative keyframes that provide a diverse and unbiased summary of the video content. This contrasts with question-relevant selection methods ($2^{nd}$ row), which may overlook general scene information. Then, KTV selects only the most informative and non-redundant visual tokens from each keyframe ($4^{th}$ row), significantly reducing spatial redundancy.
• Figure 2: Keyframes selection compared between DINO Cluster and CLIP Text-Frame similarity. CLIP selects frames that are overly dependent on the word in question, making it easy to get into "semantic traps". In contrast, DINO Cluster captures the video's maximal visual diversity, yielding a holistic and unbiased summary of its core scenes.
• Figure 3: Framework of KTV, which is a two-stage method of training-free video understanding built upon LLaVA-v1.6. First, we extract the visual features of video frames and cluster them into m clusters, whose centroids are selected as keyframes to mitigate the temporal redundancy. Second, for each frame, we select top-$k = \beta \cdot L$ key visual tokens based on their importance and redundancy, which will be fed to the LLM and prune other visual tokens to mitigate the spatial redundancy. At last, we concatenate all the remaining visual tokens and text tokens and feed them to the LLM to generate the answer.
• Figure 4: Number of visual tokens fed to the LLM and the average inference time per sample on NExTQA.
• Figure 5: Visualization of KTV using different pruning rate $\beta$. The translucent patches are the pruned visual tokens.