Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding

TL;DR

The paper tackles the problem of long-form video understanding under the computational constraints of multi-modal LLMs. It proposes Think-Clip-Sample (TCS), a training-free framework consisting of Multi-Query Reasoning to generate diverse queries and Clip-level Slow-Fast Sampling to allocate frames within informative clips while preserving global context. Empirical results on MLVU, LongVideoBench, and VideoMME show up to $6.9%$ accuracy gains and over 50% inference-time reductions on baseline MLLMs such as Qwen2-VL-7B and MiMo-VL-7B, validating both effectiveness and efficiency. The approach advances long-video understanding in resource-constrained settings and suggests future work to integrate audio and speech signals.

Abstract

Recent progress in multi-modal large language models (MLLMs) has significantly advanced video understanding. However, their performance on long-form videos remains limited by computational constraints and suboptimal frame selection. We present Think-Clip-Sample (TCS), a training-free framework that enhances long video understanding through two key components: (i) Multi-Query Reasoning, which generates multiple queries to capture complementary aspects of the question and video; and (ii) Clip-level Slow-Fast Sampling, which adaptively balances dense local details and sparse global context. Extensive experiments on MLVU, LongVideoBench, and VideoMME demonstrate that TCS consistently improves performance across different MLLMs, boosting up to 6.9% accuracy, and is capable of achieving comparable accuracy with 50% fewer inference time cost, highlighting both efficiency and efficacy of TCS on long video understanding.

Figures (4)

• Figure 1: Overview of our proposed method Think-Clip-Sample (TCS): (i) Given a question on a long video, TCS first thinks of queries from different perspectives, which are used with CLIP to retrieve frames with both high relevance and broad coverage. (ii) Instead of sampling frames with highest similarity scores, TCS identifies high-relevance clips, and then (iii) applies Slow-Fast Sampling to allocate more frames on informative clips (yellow) and distribute the remainder across non-clip regions (gray), preserving both local detail and global context.
• Figure 2: A case study on VideoMME and ablation results on Multi-Query Reasoning and Clip-level Slow-Fast Sampling component.
• Figure 3: Speed-accuracy curves of TCS and Qwen2-VL-7B.
• Figure 4: Parameter analyses on fast frame proportion and threshold $\alpha$.