Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models

TL;DR

DSR Suite tackles the lack of scalable dynamic 4D reasoning resources for vision-language models by automating the construction of DSR-Train and DSR-Bench from in-the-wild videos and 4D cues. It introduces GSM, a lightweight geometry-prior integration module built on two stacked Q-Formers, enabling selective grounding of 3D priors without harming general multimodal performance. Empirical results show state-of-the-art dynamic spatial reasoning on DSR-Bench when fine-tuning a strong base model, with robust performance preserved on general benchmarks and applicability to downstream agent tasks. This work provides a scalable framework for 4D multimodal understanding with potential impact on embodied perception and predictive world modeling in dynamic environments.

Abstract

Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.

Figures (12)

• Figure 1: Overview of DSR Suite. (a) Comparison between static and dynamic spatial reasoning: Unlike static scenarios, dynamic spatial reasoning (DSR) requires understanding environments with moving objects, posing greater reasoning challenges. (b) Data examples from DSR Suite: Built upon automated pipeline and in-the-wild videos, DSR Suite generates scalable question-answer pairs that feature viewpoint transform, multi-object interact and fine-grained answers for comprehensive training and evaluation of DSR. (c) Benchmark comparison: Evaluations on our proposed DSR-Bench highlight the capability of our model trained on constructed DSR-Train.
• Figure 2: Multiple-choice question–answer generation pipeline in our DSR Suite. It comprises three stages: Video Curation, Geometric Clue Extraction and Data Generation. In Video Curation stage, in-the-wild videos are filtered by LLMs or VLMs to remove motionless ones based on captions or visual cues. During Geometric Clue Extraction, vision foundation models extract key geometric cues, including camera poses, point clouds, object masks and orientations. Finally, in Data Generation, object coordinates are transformed into a randomly selected viewpoint and question–answer pairs are produced using either predefined templates or LLM-based free-form generation.
• Figure 3: Statistical overview of our DSR-Bench. In (a), we present the proportion of videos corresponding to different scene classes. In (b), we show the proportion of questions of various types. In (c), we depict a word cloud of the questions.
• Figure 4: Illustraction of our proposed GSM that consists of two stacked Q-Formers. The first Q-Former condenses question semantics, and the second one extracts question-relevant geometric knowledge into a compact set of geometry tokens. These tokens are appended to original vision tokens to be processed by LLM.
• Figure 5: Performance curve of accuracy on DSR-Bench with varying numbers of question-answer pairs for training.