FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting

TL;DR

FrameThinker introduces a paradigm shift for long-video understanding by enabling LVLMs to actively and iteratively interrogate video content through a multi-turn reasoning loop. It combines a two-stage training pipeline—Supervised Fine-Tuning to learn action syntax and Reinforcement Learning with a carefully designed reward scheme—and a Cognitive Consistency Verification module to maintain logical, interpretable reasoning. The method achieves state-of-the-art accuracy on challenging reasoning benchmarks while dramatically reducing the number of frames processed, e.g., 76.1% accuracy on LongVideo-Reason with only ~20.6 frames. This approach demonstrates significant practical impact for efficient, scalable video reasoning in LVLMs and offers insights into reward design and interpretability in multi-turn visual reasoning systems.

Abstract

While Large Vision-Language Models (LVLMs) have achieved substantial progress in video understanding, their application to long video reasoning is hindered by uniform frame sampling and static textual reasoning, which are inefficient and struggle to handle visually intensive video tasks. To overcome these challenges, in this paper, we introduce the concept of thinking with long videos and propose a novel framework FrameThinker. Within this framework, LVLMs are able to iteratively interrogate video content. Developing such video reasoning capabilities in LVLMs presents notable challenges, particularly in adapting the model to new video actions (e.g. select frame), and designing reward functions to guide LVLMs to adopt the newly introduced action. To solve these challenges, we propose a two-phase training strategy, first employing Supervised Fine-Tuning (SFT) to instill fundamental action capabilities, followed by Reinforcement Learning (RL) to optimize a strategic decision-making policy. Notably, in this RL phase, we conduct an in-depth and comprehensive exploration of the reward design for each action and format reward. Extensive experiments on reasoning benchmarks like Video-Holmes, LongVideo-Reason, and long-video understanding benchmarks such as LongVideoBench, MLVU, VideoMME, and LVBench, demonstrate that FrameThinker achieves a significant average improvement of +10.4% over baselines while drastically reducing the number of processed frames. Most notably, our 7B model, FrameThinker establishes a new state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average of only 20.6 frames. This not only outperforms the competitive LongVILA-R1 (72.0%) but does so with over 20x fewer frames (vs. 512), demonstrating unparalleled efficiency and effectiveness.

Figures (21)

• Figure 1: (Top) Traditional Uniform Sparse Sampling is inefficient and may miss the key frame in long videos. (Bottom) Our method starts with a sparse scan for an overview (Turn 1), then dynamically chooses frames on promising segments (Turn 2), enabling a multi-turn analysis to efficiently focus on key frames in a long video.
• Figure 2: (a) An illustration of the iterative reasoning process of our proposed FrameThiner. The model first performs a sparse scan, then uses thought-action steps to progressively gather evidence. The CCV module ensures this process is logically consistent and interpretable. (b) Our three-stage training pipeline, consisting of Data Preparation, Supervised Fine-Tuning (SFT) to learn action syntax, and Reinforcement Learning (RL) to optimize the policy.
• Figure 3: The distribution of data sources for the SFT (Left) and RL training phases (Right).
• Figure 4: Comprehensive ablation studies on key components of our training methodology. (a) Direct comparison against a fine-tuned Qwen2.5-VL-7B baseline. (b) Impact of including a format reward. (c) Ablation on the CCV module during training and inference. (d) Performance under different action reward configurations.
• Figure 5: (a) Accuracy and the number of frames processed on Video-Holmes. (b) A radar chart comparing overall performance across six benchmarks, where results are normalized and scaled for visual comparison. (c) Average accuracy across six benchmarks. Our FrameThinker achieves the best average performance.