Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesis

TL;DR

This work tackles 4D synthesis from monocular video by decomposing the problem into static 3D shape generation and dynamic motion reconstruction, anchored by an optional canonical reference mesh. It introduces Motion Latent Learning to fuse shape and video information into a motion representation and uses a frame-wise transformer to produce temporally coherent geometry; motion decoding then regresses per-frame 3D trajectories for a set of reference mesh queries, trained with a direct supervision loss. The approach achieves superior geometry and appearance fidelity on Motion-80 and Consistent4D benchmarks, demonstrates robust generalization to in-the-wild videos, and enables motion transfer to different shapes, all within a fully feed-forward pipeline. This decomposition reduces dependence on large-scale 4D data, enables efficient 4D synthesis from a single video, and provides practical benefits for generating and animating 3D assets in VR, film, and robotics contexts, with a public project page for reproducibility.

Abstract

We present Motion 3-to-4, a feed-forward framework for synthesising high-quality 4D dynamic objects from a single monocular video and an optional 3D reference mesh. While recent advances have significantly improved 2D, video, and 3D content generation, 4D synthesis remains difficult due to limited training data and the inherent ambiguity of recovering geometry and motion from a monocular viewpoint. Motion 3-to-4 addresses these challenges by decomposing 4D synthesis into static 3D shape generation and motion reconstruction. Using a canonical reference mesh, our model learns a compact motion latent representation and predicts per-frame vertex trajectories to recover complete, temporally coherent geometry. A scalable frame-wise transformer further enables robustness to varying sequence lengths. Evaluations on both standard benchmarks and a new dataset with accurate ground-truth geometry show that Motion 3-to-4 delivers superior fidelity and spatial consistency compared to prior work. Project page is available at https://motion3-to-4.github.io/.

Figures (12)

• Figure 1: From a single glance, Motion 3-to-4 unfolds: weaving time, shape, and movement into living 4D reality.
• Figure 2: An overview of our Motion 3-to-4 framework for 4D synthesis. At the core of the framework is a motion–latent learning module consisting of a geometry encoder and a video encoder, which jointly process the input video and sampled points. The resulting latent tokens are decoded into a frame-wise 3D motion flow relative to the first video frame, producing temporally consistent 4D assets.
• Figure 3: Geometric comparison on the Consistent4D benchmark Consistent4d. Through spatially consistent motion reconstruction, we obtain plausible and high-quality 3D geometry.
• Figure 4: Qualitative Comparisons. We compare our method with strong baselines including GVFD gvfd, L4GM L4gm, and V2M4 V2M4 on our proposed Motion-80 benchmark. For fair evaluation, we render the generated 4D assets from all methods into an orthogonal novel view. Our approach produces more temporally coherent and structurally consistent motion. We invite reviewers to consult the supplemental material for animation visualization.
• Figure 5: In-the-Wild Video-to-4D Synthesis. Our method generalizes to diverse in-the-wild inputs, including real-world videos (top row) and generated animations (bottom row). By formulating motion reconstruction as surface-to-pixel alignment, we achieve robust local correspondence reasoning across varied shapes and motion patterns.