How Much 3D Do Video Foundation Models Encode?

TL;DR

This work introduces a model-agnostic framework to quantify 3D awareness in VidFMs by training a shallow readout to predict dense 3D points, depth, and camera poses from frozen video features. It demonstrates that state-of-the-art video generators exhibit strong 3D understanding, often rivaling domain-specific 3D experts, and that temporal reasoning is a key driver of this capability. The study further analyzes how 3D fine-tuning, model scale, and diffusion-layer choices affect 3D awareness, revealing nuanced gains and potential generalization trade-offs. Additionally, it shows that VidFM features can power feedforward 3D reconstruction effectively, especially in low-data regimes, and provides a benchmark and protocol for assessing 3D properties across VidFMs.

Abstract

Videos are continuous 2D projections of 3D worlds. After training on large video data, will global 3D understanding naturally emerge? We study this by quantifying the 3D understanding of existing Video Foundation Models (VidFMs) pretrained on vast video data. We propose the first model-agnostic framework that measures the 3D awareness of various VidFMs by estimating multiple 3D properties from their features via shallow read-outs. Our study presents meaningful findings regarding the 3D awareness of VidFMs on multiple axes. In particular, we show that state-of-the-art video generation models exhibit a strong understanding of 3D objects and scenes, despite not being trained on any 3D data. Such understanding can even surpass that of large expert models specifically trained for 3D tasks. Our findings, together with the 3D benchmarking of major VidFMs, provide valuable observations for building scalable 3D models.

Figures (7)

• Figure 1: We study the emergence of 3D in video foundation models by probing their features with 3D reconstruction tasks. Our study reveals state-of-the-art video generators develop strong 3D understanding even compared to 3D experts, despite only trained on 2D data.
• Figure 2: Overview of the Probe. We extract video features using various video foundation models and keep the features frozen. We sample four frames from the original video clip and fetch the corresponding feature maps from the video features. We train the probe by taking these spatial features as input, and task the probe to estimate point maps, depth maps and camera poses. Our probe model consists of a shallow transformer and three readout heads. We measure the estimation errors as the main indicators of 3D awareness.
• Figure 3: CO3Dv2 qualitative results. For each scene, we show input frames and the unprojected 3D points prediction. Fast3R, WAN2.1-14B, and Open-Sora2.0 best preserve intricate details (e.g., the truck’s gripper) and reconstruct the overall structure.
• Figure 4: DL3DV qualitative results. On this more challenging dataset, DINOv2 sometimes fails catastrophically. Top video generators often retain coherent geometry, where WAN2.1-14B produces the sharpest and most accurate point clouds overall.
• Figure 5: Layer--timestep ablations. We show point-map error (lower is better) on the ablation data when probing different diffusion layers and denoising time steps. Best results are consistently from mid layers and early-but-not-first time steps.