C4D: 4D Made from 3D through Dual Correspondences

TL;DR

This work addresses 4D reconstruction of dynamic scenes from monocular video by extending dense 3D pointmap reconstruction with dual temporal correspondences: short-term optical flow and long-term point tracking. It introduces a Dynamic-aware Point Tracker (DynPT) to predict point mobility and uses correspondence-guided motion masks to focus static-background reconstruction, complemented by CMA, CTS, and a novel PTS objective to ensure smooth per-frame geometry and trajectories. The resulting C4D framework delivers complete 4D recovery, improving depth and camera pose estimation, and providing robust point tracking and motion masks across challenging dynamic datasets. The approach demonstrates strong performance on multiple downstream tasks and leverages pretrained 3D weights with targeted correspondence-driven optimization for practical dynamic-scene 4D reconstruction from monocular video.

Abstract

Recovering 4D from monocular video, which jointly estimates dynamic geometry and camera poses, is an inevitably challenging problem. While recent pointmap-based 3D reconstruction methods (e.g., DUSt3R) have made great progress in reconstructing static scenes, directly applying them to dynamic scenes leads to inaccurate results. This discrepancy arises because moving objects violate multi-view geometric constraints, disrupting the reconstruction. To address this, we introduce C4D, a framework that leverages temporal Correspondences to extend existing 3D reconstruction formulation to 4D. Specifically, apart from predicting pointmaps, C4D captures two types of correspondences: short-term optical flow and long-term point tracking. We train a dynamic-aware point tracker that provides additional mobility information, facilitating the estimation of motion masks to separate moving elements from the static background, thus offering more reliable guidance for dynamic scenes. Furthermore, we introduce a set of dynamic scene optimization objectives to recover per-frame 3D geometry and camera parameters. Simultaneously, the correspondences lift 2D trajectories into smooth 3D trajectories, enabling fully integrated 4D reconstruction. Experiments show that our framework achieves complete 4D recovery and demonstrates strong performance across multiple downstream tasks, including depth estimation, camera pose estimation, and point tracking. Project Page: https://littlepure2333.github.io/C4D

Figures (9)

• Figure 1: Given a monocular video that contains both camera movement and object movement, C4D can recover the dynamic scene in 4D, including per-frame dense point cloud, camera poses and intrinsic parameters. Video depth, motion masks, and point tracking in both 2D and 3D space are also available in the outputs.
• Figure 2: Overview of C4D. C4D takes monocular video as input and jointly predicts dense 3D pointmaps (Sec. \ref{['sec:3d']}) and temporal correspondences (Sec. \ref{['sec:correspondence']}), including short-term optical flow and long-term point tracking (Sec. \ref{['sec:track']}). These correspondences are utilized to predict motion masks (Sec. \ref{['sec:motion_mask']}) and participate in the optimization process (Sec. \ref{['sec:optimization']}) with 3D pointmaps to obtain 4D outputs.
• Figure 3: Architecture of Dynamic-aware Point Tracker (DynPT). For given video input and sampled initial query points, DynPT uses Transformer to iteratively update the tracks with features obtained from both 3D-aware ViT encoder and CNN.
• Figure 4: Correspondence-guided motion mask prediction. The solid circle indicates predicted dynamic points, the hollow circle indicates predicted static points. Adjacent frames are from constructed image pairs containing the current frame.
• Figure 5: Ablation illustration of Point Trajectory Smoothness (PTS) objective. The temporal depth and 3D trajectories become more smooth after applying PTS objective.