From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding

TL;DR

Long-form video understanding is hindered by excessive visual tokens that exceed language model context windows. The paper introduces Frames-to-Clips (F2C), a training-free framework that selects anchor key frames and expands them into temporally coherent key clips with adaptive resolution under a fixed token budget, balancing clip length and spatial detail. Empirical results on Video-MME, LongVideoBench, and MLVU show that F2C consistently surpasses uniform sampling and other baselines, with notable gains in motion-sensitive and reasoning tasks, and improved token efficiency. This approach provides a practical, scalable pathway to leverage Video LLMs for real-world long-form video tasks without requiring model retraining.

Abstract

Video Large Language Models (VLMs) have achieved strong performance on various vision-language tasks, yet their practical use is limited by the massive number of visual tokens produced from raw video frames, which quickly exhausts the model's context window. Existing solutions mitigate this issue by selecting a sparse set of frames, but such frame-wise selection discards essential temporal dynamics in long-form videos, leading to suboptimal reasoning about motion and event continuity. In this work, we systematically examine the role of temporal information and show that extending selection from isolated key frames to temporally coherent key clips improves video understanding. To maintain a fixed computational budget while accommodating the larger token footprint of clips, we introduce frame resolution as a controllable factor in frame selection, enabling a trade-off between spatial resolution and clip length. Building on this idea, we propose an adaptive clip length module that dynamically balances these factors to ensure a constant token count per video. Experiments on three long-form video benchmarks demonstrate that our training-free approach, F2C, outperforms uniform sampling by up to 8.1%, 5.6%, and 10.3% on Video-MME, LongVideoBench, and MLVU, respectively. These results highlight the importance of preserving temporal coherence in frame selection and provide a practical pathway for scaling VLMs to real-world video understanding applications. Project webpage is available at https://guangyusun.com/f2c .

Figures (13)

• Figure 1: Motivation of key clip selection. In long-form videos (e.g., 30 minutes), uniform sampling into 32 frames yields gaps of more than 50 seconds, so only a single frame may be selected from an event. As an example from shangguan2025tomatoassessingvisualtemporal shown, relying on one key frame (a) makes it impossible to answer motion-related questions such as the rotation direction of a Ping Pong ball. By extending a key frame into a short key clip (b), temporal continuity is preserved, enabling the correct reasoning.
• Figure 2: Uniform Sampling samples frames evenly from the video, and Key Frames are human-annotated crucial frames to answer the question. Uniform and Key Clips augment frames with their neighboring frames.
• Figure 3: Overview of F2C. A video and text query are first processed by a vision-language encoder to select (a) anchor key frames. Each anchor is then extended into a short key clip through (b) adaptive clip length selection, which adjusts the trade-off between resolution and clip length under the same budget. The constructed key clips preserve temporal continuity and, together with the query, are fed into a VLM to generate the answer.
• Figure 4: Performance on each type of questions on Video-MME with 16 frames.
• Figure 5: Impact of different initial keyframe selectors on Video-MME with $K=16$ and extending them to key clips by F2C.