ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos

TL;DR

The paper investigates whether multimodal large language models can acquire human-like egocentric 4D spatial-temporal reasoning from video input. It introduces Ego-ST Bench, a dataset with over 5,000 QA pairs across eight spatial-temporal tasks, including forward and reverse reasoning, and ST-R1, a two-stage video reasoning model trained with Chain-of-Thought supervised fine-tuning and Group Relative Policy Optimization. ST-R1 demonstrates substantial gains from limited high-quality data and narrows gaps between open-source and closed-source systems, with notable improvements on out-of-distribution tasks (e.g., over $32\%$). Together, Ego-ST Bench and ST-R1 provide a practical benchmark and training framework to advance video-based spatial-temporal reasoning in egocentric settings.

Abstract

Humans excel at spatial-temporal reasoning, effortlessly interpreting dynamic visual events from an egocentric viewpoint. However, whether multimodal large language models (MLLMs) can similarly understand the 4D world remains uncertain. This paper explores multimodal spatial-temporal reasoning from an egocentric perspective, aiming to equip MLLMs with human-like reasoning capabilities. To support this objective, we introduce \textbf{Ego-ST Bench}, a novel benchmark containing over 5,000 question-answer pairs across four categories, systematically evaluating spatial, temporal, and integrated spatial-temporal reasoning. Additionally, we propose \textbf{ST-R1} training paradigm, a video-based reasoning model that incorporates reverse thinking into its reinforcement learning process, significantly enhancing performance. We combine long-chain-of-thought (long-CoT) supervised fine-tuning with Group Relative Policy Optimization (GRPO) reinforcement learning, achieving notable improvements with limited high-quality data. Ego-ST Bench and ST-R1 provide valuable insights and resources for advancing video-based spatial-temporal reasoning research.

Figures (8)

• Figure 1: Top: The Ego-ST bench is characterized by Egocentric Spatio-Temporal reasoning questions, and two spatial-temporal reasoning tasks are chosen here as examples: Route Description and Direction change selection. Bottom left: Shows the performance of each model in a spatial-temporal reasoning task. Bottom right: Shows the results of the 4 training methods compared to baseline. (In the radar chart, F stands for forward and R stands for reverse.)
• Figure 2: Benchmark pipeline. The pipeline starts by manually filtering segments from different datasets and self-collected video data into a standardised format for consistent processing. Forward and reverse route description QA pairs are then generated through manual annotations and question templates. After obtaining the manually annotated route description QA pairs GPT-4o api is used to generate the corresponding three types of multi-selected QA pairs.
• Figure 3: Demonstration of tasks for Ego-ST bench. It includes 8 QAs for 4 types of tasks, which are route description, direction description, landmark description and action description.
• Figure 4: Benchmark Statistics. Left: The distribution of tasks across four main categories. Right: The video length statistic.
• Figure 5: Spatial Temporal Reasoning Model. Our model is trained in two stages: (1) Create Chain of Thought (CoT) data for supervised fine-tuning (SFT). (2) Enhancing the model using the rule-based reinforcement learning GRPO algorithm.