Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video

TL;DR

Mesh4D tackles monocular 4D mesh reconstruction by integrating a pretrained static 3D generator with a deformation VAE and a diffusion model to predict a full 4D mesh sequence from a single video. The deformation VAE encodes the object's motion across the entire sequence into a compact latent $\boldsymbol{z}^d$, guided by skeleton priors during training and enhanced by a spatio-temporal transformer to learn coherent temporal dynamics; skeleton information is not required at inference. A diffusion model conditioned on the canonical first-frame mesh $\mathcal{M}_1$ and the input video $\mathcal{I}$ generates the deformation latent, enabling a single-pass reconstruction of $\mathcal{M}_t$ for all times $t$. On a synthetic Objaverse-based benchmark, Mesh4D achieves state-of-the-art geometry, dense tracking, and novel-view synthesis, with ablations confirming the benefits of skeleton guidance and spatio-temporal attention. Limitations include topology changes and reliance on high-quality canonical meshes and skeletons during training, but the approach offers a scalable pathway to complete 4D reconstructions from monocular input.

Abstract

We propose Mesh4D, a feed-forward model for monocular 4D mesh reconstruction. Given a monocular video of a dynamic object, our model reconstructs the object's complete 3D shape and motion, represented as a deformation field. Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass. This latent space is learned by an autoencoder that, during training, is guided by the skeletal structure of the training objects, providing strong priors on plausible deformations. Crucially, skeletal information is not required at inference time. The encoder employs spatio-temporal attention, yielding a more stable representation of the object's overall deformation. Building on this representation, we train a latent diffusion model that, conditioned on the input video and the mesh reconstructed from the first frame, predicts the full animation in one shot. We evaluate Mesh4D on reconstruction and novel view synthesis benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.

Figures (8)

• Figure 1: Illustration of Mesh4D. Given a monocular RGB video as input, Mesh4D generates a complete animated 3D mesh and its deformation. Each 4D reconstruction is shown at several time steps, the top layer displaying normals and the bottom one textured meshes.
• Figure 2: Overall Deformation VAE pipeline. (Left) Given a sequence of 3D meshes as input, we first uniformly sample a sequence of corresponding points. We inject the skeleton information by using masked self- and cross-attention. Then, a Farthest Point Sampling (FPS) at spatial dimension is performed to compress the latent, followed by 8 layers of spatio-temporal attention. The deformation field is decoded by layers of spatio-temporal attention, followed by a cross attention where canonical vertices serve as query points. (Right) Each of our spatio-temporal attention layers sequentially performs temporal attention, global attention, and spatial attention. For temporal and global attention, we additionally apply 1D RoPE DBLP:journals/corr/abs-2104-09864 embedding on the temporal dimension.
• Figure 3: Overall deformation diffusion model pipeline. We build it based on HY3D 2.1 hunyuan3d2025hunyuan3d21 shape diffusion model with additional spatial and temporal embedding as well as cross attention layer to condition the deformation field generation on the canonical mesh and input video.
• Figure 4: Qualitative results on geometry reconstruction. We show both the normal map and the error map (the bluer the better). HY3D 2.1 hunyuan3d2025hunyuan3d21 suffers from inaccurate pose and shape estimation due to the lack of temporal information. Thanks to the spatio-temporal attention, our method manages to reconstruct the mesh that follows the given input frames with accurate pose and similar shape.
• Figure 5: Qualitative results on novel view synthesis. All the state-of-the-art methods suffer from inaccurate pose estimation, either due to lack of temporal attention (HY3D hunyuan3d2025hunyuan3d21) or neglect the importance of geometric supervision (GVDF zhang2025GVFD, L4GM ren2024l4gm). 3D-GS based methods occasionally exhibit ghost artifacts because they lack topology constraints during deformation, while the frame-wise reconstruction method produce inconsistent shape and texture. Moreover, by leveraging a large reconstruction method, we avoid predicting extremely incorrect canonical mesh. Thanks to the skeleton information and spatio-temporal attention, Mesh4D is able to reconstruct accurate pose and geometry, and produces temporally consistent novel view video.