Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning

TL;DR

Video-o3 tackles long-form video understanding by enabling native interleaved clue seeking and multi-turn reasoning. It introduces Task-Decoupled Attention Masking to prevent attention dilution and Verifiable Trajectory-Guided Reward to balance exploration with efficiency, trained with a large Seeker-173K trajectory dataset. The approach achieves state-of-the-art results on multiple long-video benchmarks and demonstrates efficient, multi-hop evidence gathering via adaptive tool usage. The work advances end-to-end training for long-video reasoning and highlights practical pathways for scalable, tool-enabled video intelligence.

Abstract

Existing multimodal large language models for long-video understanding predominantly rely on uniform sampling and single-turn inference, limiting their ability to identify sparse yet critical evidence amid extensive redundancy. We introduce Video-o3, a novel framework that supports iterative discovery of salient visual clues, fine-grained inspection of key segments, and adaptive termination once sufficient evidence is acquired. Technically, we address two core challenges in interleaved tool invocation. First, to mitigate attention dispersion induced by the heterogeneity of reasoning and tool-calling, we propose Task-Decoupled Attention Masking, which isolates per-step concentration while preserving shared global context. Second, to control context length growth in multi-turn interactions, we introduce a Verifiable Trajectory-Guided Reward that balances exploration coverage with reasoning efficiency. To support training at scale, we further develop a data synthesis pipeline and construct Seeker-173K, comprising 173K high-quality tool-interaction trajectories for effective supervised and reinforcement learning. Extensive experiments show that Video-o3 substantially outperforms state-of-the-art methods, achieving 72.1% accuracy on MLVU and 46.5% on Video-Holmes. These results demonstrate Video-o3's strong multi-hop evidence-seeking and reasoning capabilities, and validate the effectiveness of native tool invocation in long-video scenarios.

Figures (15)

• Figure 1: Overview of our proposed . Guided by the query and current visual observations, actively identifies and localizes critical visual clues. It utilizes native interleaved tool invocation to capture video clips with dynamic quota. Following a detailed scrutiny of these local segments, the model autonomously decides whether to continue the search for further evidence or to conclude the reasoning process with a direct answer.
• Figure 2: Schematic comparison of different reasoning paradigms. (a) Conventional Reasoning: The MLLM attempts to derive the answer directly from the initial input without intermediate exploration. (b) Decoupled Iterative Reasoning: The model performs multi-step reasoning via independent calls. Crucially, the context is reset or isolated between turns. (c) Native Interleaved Tool Invocation (Ours): It executes multi-hop reasoning within a unified shared context. This design enables the preservation of both visual features and reasoning history across interleaved turns, facilitating holistic joint inference.
• Figure 3: Overview of our proposed . It dynamically executes tool invocations based on previously reasoning to scrutinize specific clue segments via VideoCrop with adaptive spatiotemporal resolution. When the accumulated clues are sufficient to answer the question, is able to terminate the search process and combine multiple clue segments to derive the correct answer.
• Figure 4: Illustration of Task-Decoupled Attention Masking. The heatmap illustrates the visibility constraints imposed during the multi-turn supervised fine-tuning process. The global overview is masked during the Answer phase to prevent fake thinking, while local tool outputs are masked during Tool Call planning to force reliance on global view.
• Figure 5: Overview of the data construction pipeline. We initiate the process by curating high-quality "Video-Question-Answer" database. These inputs are then transformed into explicit tool exploration trajectories via a rigorous four-stage annotation pipeline. Human verification is enforced through random sampling in all stages.