SpatialVID: A Large-Scale Video Dataset with Spatial Annotations

TL;DR

SpatialVID introduces a large-scale real-world video dataset with dense geometric and semantic annotations to bridge 3D reconstruction and world simulation. By curating 21,000+ hours into 7,089 hours of dynamic content and a 1,111-hour SpatialVID-HQ subset, and by generating per-frame camera poses, depth maps, dynamic masks, motion instructions, and structured captions, the work enables robust spatial learning at scale. The authors validate SpatialVID across camera-controlled video generation, novel view synthesis, and geometric prediction, showing improvements over prior datasets like Panda-70M in controllability, realism, and 3D generalization. They acknowledge limitations in pose estimation under extreme motion and call for future advances (e.g., ViPE) to further enhance robustness. Overall, SpatialVID serves as a scalable, richly annotated resource for advancing spatial intelligence in video understanding and synthesis.

Abstract

Significant progress has been made in spatial intelligence, spanning both spatial reconstruction and world exploration. However, the scalability and real-world fidelity of current models remain severely constrained by the scarcity of large-scale, high-quality training data. While several datasets provide camera pose information, they are typically limited in scale, diversity, and annotation richness, particularly for real-world dynamic scenes with ground-truth camera motion. To this end, we collect SpatialVID, a dataset consists of a large corpus of in-the-wild videos with diverse scenes, camera movements and dense 3D annotations such as per-frame camera poses, depth, and motion instructions. Specifically, we collect more than 21,000 hours of raw videos, and process them into 2.7 million clips through a hierarchical filtering pipeline, totaling 7,089 hours of dynamic content. A subsequent annotation pipeline enriches these clips with detailed spatial and semantic information, including camera poses, depth maps, dynamic masks, structured captions, and serialized motion instructions. Analysis of SpatialVID's data statistics reveals a richness and diversity that directly fosters improved model generalization and performance, establishing it as a key asset for the video and 3D vision research community.

Figures (21)

• Figure 1: We introduce SpatialVID, a large-scale video dataset with explicit spatial annotations including camera poses, depth maps, structured captions and serialized motion instructions. The dataset consists of 7,089 hours of real-world dynamic scenes. An exemplar of our SpatialVID is shown on the right.
• Figure 2: Overview of the curation pipeline. The pipeline comprises three stages: filtering, annotation, and sampling. We start from manually collected web videos with notable camera motion. In the filtering stage, raw videos are hierarchically preprocessed and filtered. The annotation stage adds geometric and semantic labels and derives motion instructions from camera poses. The sampling stage then balances clips by motion and category to form a high-quality subset (SpatialVID-HQ) with well-distributed classes for downstream tasks.
• Figure 3: Structured caption generation. A VLM produces initial motion and scene descriptions, which the LLM refines using camera poses to yield structured captions.
• Figure 4: Effect of spatial enhancement. After applying spatial enhancement, the LLM corrected the incorrect direction (right) output by the VLM to left.
• Figure 5: Dataset quality comparison. Comparison of SpatialVID, its balanced subset (SpatialVID-HQ), and the Panda70M-test set processed with the same pipeline. Histograms and KDE curves reveal distribution patterns across quality metrics, showing that SpatialVID-HQ achieves consistently superior quality, validating our manual collection, filtering, and sampling process.