V-DPM: 4D Video Reconstruction with Dynamic Point Maps

TL;DR

This work shows how to formulate DPMs for video input in a way that maximizes representational power, facilitates neural prediction, and enables reuse of pretrained models, and implements these ideas on top of VGGT, a recent and powerful 3D reconstructor.

Abstract

Powerful 3D representations such as DUSt3R invariant point maps, which encode 3D shape and camera parameters, have significantly advanced feed forward 3D reconstruction. While point maps assume static scenes, Dynamic Point Maps (DPMs) extend this concept to dynamic 3D content by additionally representing scene motion. However, existing DPMs are limited to image pairs and, like DUSt3R, require post processing via optimization when more than two views are involved. We argue that DPMs are more useful when applied to videos and introduce V-DPM to demonstrate this. First, we show how to formulate DPMs for video input in a way that maximizes representational power, facilitates neural prediction, and enables reuse of pretrained models. Second, we implement these ideas on top of VGGT, a recent and powerful 3D reconstructor. Although VGGT was trained on static scenes, we show that a modest amount of synthetic data is sufficient to adapt it into an effective V-DPM predictor. Our approach achieves state of the art performance in 3D and 4D reconstruction for dynamic scenes. In particular, unlike recent dynamic extensions of VGGT such as P3, DPMs recover not only dynamic depth but also the full 3D motion of every point in the scene.

Figures (7)

• Figure 1: V-DPM results. We propose a method for extending state-of-the-art static 3D reconstructors like VGGT with Dynamic Point Maps (DPMs). Given a video snippet, V-DPM reconstructs the 3D motion of the scene (i.e., the scene flow), along with its 3D shape and the camera parameters. Because of DPMs, the same representation captures both the static background and complex non-rigid motion.
• Figure 2: Model architecture of V-DPM. Our model decodes both time-variant point maps as in MonST3R zhang24monst3r: and time-invariant point maps corresponding to a fixed timestamp $t_j$ via the proposed time-conditioned decoder.
• Figure 3: V-DPM point maps. The point maps $\mathcal{P}$ (yellow) are time-variant: they predict the 3D points at their respective input timestamps (we do not show the argument $\pi_0$ for compactness). The point maps $\mathcal{Q}$ (green) are time-invariant: they predict the 3D points at a common reference timestamp $t_j$.
• Figure 4: Transformer block in the time-conditioned decoder. Conditioning is implemented via adaptive LayerNorm perez18film:peebles23scalable.
• Figure 5: Dynamic point maps of a robot doing a manipulation task.