Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence

TL;DR

Open-o3 Video tackles the challenge of grounded video reasoning by generating explicit spatio-temporal evidence—timestamped frames and bounding boxes—to support answers. It introduces two curated datasets, STGR-CoT-30k for supervised fine-tuning and STGR-RL-36k for reinforcement learning, and adopts a two-stage training regime with adaptive temporal proximity and temporal gating via GSPO to achieve coherent temporal localization and spatial grounding. On V-STAR, it sets a new state of the art, with substantial gains over strong baselines, and shows consistent improvements across a broad range of video understanding benchmarks, while enabling confidence-aware test-time scaling using reasoning traces. These contributions advance interpretable, evidence-centered video reasoning and offer scalable, verifiable inference for complex dynamic scenes with potential extensions to longer videos and multimodal signals including audio.

Abstract

Most video reasoning models only generate textual reasoning traces without indicating when and where key evidence appears. Recent models such as OpenAI-o3 have sparked wide interest in evidence-centered reasoning for images, yet extending this ability to videos is more challenging, as it requires joint temporal tracking and spatial localization across dynamic scenes. We introduce Open-o3 Video, a non-agent framework that integrates explicit spatio-temporal evidence into video reasoning, and carefully collect training data and design training strategies to address the aforementioned challenges. The model highlights key timestamps, objects, and bounding boxes alongside its answers, allowing reasoning to be grounded in concrete visual observations. To enable this functionality, we first curate and build two high-quality datasets, STGR-CoT-30k for SFT and STGR-RL-36k for RL, with carefully constructed temporal and spatial annotations, since most existing datasets offer either temporal spans for videos or spatial boxes on images, lacking unified spatio-temporal supervision and reasoning traces. Then, we adopt a cold-start reinforcement learning strategy with multiple specially designed rewards that jointly encourage answer accuracy, temporal alignment, and spatial precision. On V-STAR benchmark, Open-o3 Video achieves state-of-the-art performance, raising mAM by 14.4% and mLGM by 24.2% on the Qwen2.5-VL baseline. Consistent improvements are also observed on a broad range of video understanding benchmarks, including VideoMME, WorldSense, VideoMMMU, and TVGBench. Beyond accuracy, the reasoning traces produced by Open-o3 Video also provide valuable signals for test-time scaling, enabling confidence-aware verification and improving answer reliability.

Figures (9)

• Figure 1: While prior video reasoning models (e.g., Video-R1 video-r1, VideoRFT wang2025videorft) only generate textual rationales, Open-o3 Video integrates explicit spatio-temporal grounding into the reasoning process. The model highlights key timestamps and object regions that directly support the answer, providing verifiable evidence for its prediction. More visualizations are provided in Appendix \ref{['sec:ap_vis']}.
• Figure 2: Overview of our data construction pipeline and dataset composition. Left: The annotation pipeline includes Gemini 2.5 Pro initial annotation, bounding box filtering, and self-consistency checking. Right: Distribution of data categories in STGR-CoT-30k (SFT) and STGR-RL-36k (RL), showing a balanced coverage across temporal, spatial, spatio-temporal, and general QA.
• Figure 3: Overview of Open-o3 Video. We adopt a two-stage training paradigm: (a) cold-start initialization to learn structured, grounded outputs; (b) reinforcement learning with a composite reward that sharpens temporal alignment and spatial precision with adaptive temporal proximity and temporal gating.
• Figure 4: Annotation Prompt for PLM-Video-Human Region Dense Temporal Captioning Data source.
• Figure 5: Annotation Prompt for Temporal Grounding Data Source.