Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting

TL;DR

Vivid4D tackles monocular 4D reconstruction by marrying geometric priors with video diffusion-based view augmentation. It reframes view augmentation as a video inpainting task trained on unposed web videos, using 2D tracking to synthesize occlusion masks and an anchor-conditioned diffusion model to complete novel views. An iterative view augmentation strategy expands viewpoint coverage while depth-scale alignment and a robust RGB loss (IV) mitigate artifacts from depth inaccuracies. Empirical results on iPhone and HyperNeRF data show improved multi-view consistency, foreground detail preservation, and filled unseen regions, highlighting the value of integrating geometry with generative priors for dynamic scene reconstruction from casual videos.

Abstract

Reconstructing 4D dynamic scenes from casually captured monocular videos is valuable but highly challenging, as each timestamp is observed from a single viewpoint. We introduce Vivid4D, a novel approach that enhances 4D monocular video synthesis by augmenting observation views - synthesizing multi-view videos from a monocular input. Unlike existing methods that either solely leverage geometric priors for supervision or use generative priors while overlooking geometry, we integrate both. This reformulates view augmentation as a video inpainting task, where observed views are warped into new viewpoints based on monocular depth priors. To achieve this, we train a video inpainting model on unposed web videos with synthetically generated masks that mimic warping occlusions, ensuring spatially and temporally consistent completion of missing regions. To further mitigate inaccuracies in monocular depth priors, we introduce an iterative view augmentation strategy and a robust reconstruction loss. Experiments demonstrate that our method effectively improves monocular 4D scene reconstruction and completion. See our project page: https://xdimlab.github.io/Vivid4D/.

Figures (12)

• Figure 1: Vivid4D. We improve dynamic scene reconstruction from casually captured monocular videos by synthesizing augmented views. Our approach integrates both geometric and generative priors to reformulate the video augmentation as a video inpainting task. This enables our method to effectively complete invisible regions in the scene and enhance reconstruction quality.
• Figure 2: Video Inpainting for 4D Reconstruction. To train the video inpainting model, we use 2D tracking to generate masked training pairs from unposed web videos. During 4D reconstruction, we warp the monocular video to novel viewpoints, creating masked videos that our inpainting diffusion model then completes.
• Figure 3: 4D reconstruction based on view augmentation. Given an input monocular video, we first perform sparse reconstruction to obtain camera poses and align monocular depth to metric scale, forming an initial data buffer $\mathcal{D}_0$. In each iterative view augmentation step, we select frames at each timestamp from the previous buffer $\mathcal{D}_{j-1}$ and warp them to novel viewpoints using pre-defined camera poses $\mathbf{T}$, creating new perspective videos with continuous invisible region masks. These masked videos, along with binary masks and an anchor video, are fed into our pre-trained anchor-conditioned video inpainting diffusion model to produce completed novel-view videos. We update the buffer $\mathcal{D}_j$ with these enhanced videos, their metric depths and poses. Finally, both the original monocular video and all synthesized multi-view videos are used to supervise 4D scene reconstruction.
• Figure 4: Difference between Direct Warping and Iterative Warping.$\mathcal{S}$ and $\mathcal{T}$ denote spatial and temporal dimensions, with $\mathcal{V}$ being the input monocular video (yellow dots indicating camera poses in 4D space). Left: With $N=1$, we directly warp the input video to new perspectives (green dots) in a single iteration. Warped frames are ranked by warping distance per timestamp and organized into a video (purple underline). Right: With $N=2$, the video is first warped the closest pre-defined poses (green dots, $j=1$), then additional perspectives (red dots, $j=2$) are generated by selecting frames with minimal warping angles from existing frames. This iterative approach minimizes distortion and floaters caused by depth inaccuracies in large-angle warping, enhancing reconstruction quality.
• Figure 5: Qualitative comparison of dynamic scene reconstruction on iPhone dataset and HyperNeRF dataset. The black holes in 4D GS and the white areas in Shape of Motion indicate regions where the input video lacks visibility. In contrast, our method effectively fills these invisible areas within the scenes, leveraging multi-view constraints and spatiotemporal priors to enhance reconstruction quality.