Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos

TL;DR

The paper tackles reconstructing dynamic 3D scenes from unsynchronized and uncalibrated multi-view videos by leveraging human motion as a calibration signal. It first estimates temporal offsets and camera poses from SMPL-based human motion using DTW and Procrustes alignment, then jointly optimizes a 4D NeRF (K-Planes) with a progressive, coarse-to-fine training schedule. Empirical results on real-world datasets show robust initialization, precise calibration after refinement, and high-quality dynamic reconstructions, outperforming Sync-NeRF in dynamic settings. The work broadens the applicability of dynamic NeRF to casual captures and moving-camera scenarios, with potential extensions to non-human objects and more complex occlusion handling.

Abstract

Recent works on dynamic 3D neural field reconstruction assume the input from synchronized multi-view videos whose poses are known. The input constraints are often not satisfied in real-world setups, making the approach impractical. We show that unsynchronized videos from unknown poses can generate dynamic neural fields as long as the videos capture human motion. Humans are one of the most common dynamic subjects captured in videos, and their shapes and poses can be estimated using state-of-the-art libraries. While noisy, the estimated human shape and pose parameters provide a decent initialization point to start the highly non-convex and under-constrained problem of training a consistent dynamic neural representation. Given the shape and pose parameters of humans in individual frames, we formulate methods to calculate the time offsets between videos, followed by camera pose estimations that analyze the 3D joint positions. Then, we train the dynamic neural fields employing multiresolution grids while we concurrently refine both time offsets and camera poses. The setup still involves optimizing many parameters; therefore, we introduce a robust progressive learning strategy to stabilize the process. Experiments show that our approach achieves accurate spatio-temporal calibration and high-quality scene reconstruction in challenging conditions.

Figures (12)

• Figure 1: We propose an approach to reconstruct dynamic 3D scenes from unsynchronized and uncalibrated videos. We exploit human motion as a calibration pattern to align time offsets and camera poses.
• Figure 2: Both transformer-based method DUSt3R dust3r and SfM library COLMAP colmap fail to recover correct camera poses.
• Figure 3: Overview of our method. Given unsynchronized multi-view videos without camera poses, we first extract human motion independently. Then we estimate (a) time offsets and (b) global camera poses by aligning human motions. Starting from the initial point, (c) we further refine both camera poses and time offsets by jointly optimizing them with dynamic NeRF with progressive training.
• Figure 4: Examples of estimated human motion on our dataset. While SLAHMR slahmr can reconstruct valid human motion for most frames (left), it sometimes fails when there is ambiguity due to the viewing angle (middle) or fast motion (right).
• Figure 5: We visualize estimated camera poses of the initialization step (top) and after refinement (bottom) on various datasets. Blue and red frustums are estimation and ground truth, respectively.