SirenPose: Dynamic Scene Reconstruction via Geometric Supervision

TL;DR

SirenPose tackles dynamic 4D scene reconstruction from monocular video by introducing a geometry-aware loss that fuses high-frequency SIREN modeling with keypoint-based geometric priors. It decomposes spatiotemporal signals into low-frequency global structure and high-frequency dynamics, guided by CAPE-style priors and graph-based keypoint relationships, and enforces structural coherence through a sine-space geometric loss. The approach yields state-of-the-art results on Sintel, Bonn, and DAVIS, improving FVD, FID, LPIPS, and pose accuracy, while enhancing temporal coherence; ablations confirm the complementary roles of high/low-frequency supervision and geometric priors. Overall, SirenPose provides a versatile loss function that improves detail fidelity, physical plausibility, and stability in monocular dynamic 4D reconstructions, with strong generalization across frameworks and datasets.

Abstract

We introduce SirenPose, a geometry-aware loss formulation that integrates the periodic activation properties of sinusoidal representation networks with keypoint-based geometric supervision, enabling accurate and temporally consistent reconstruction of dynamic 3D scenes from monocular videos. Existing approaches often struggle with motion fidelity and spatiotemporal coherence in challenging settings involving fast motion, multi-object interaction, occlusion, and rapid scene changes. SirenPose incorporates physics inspired constraints to enforce coherent keypoint predictions across both spatial and temporal dimensions, while leveraging high frequency signal modeling to capture fine grained geometric details. We further expand the UniKPT dataset to 600,000 annotated instances and integrate graph neural networks to model keypoint relationships and structural correlations. Extensive experiments on benchmarks including Sintel, Bonn, and DAVIS demonstrate that SirenPose consistently outperforms state-of-the-art methods. On DAVIS, SirenPose achieves a 17.8 percent reduction in FVD, a 28.7 percent reduction in FID, and a 6.0 percent improvement in LPIPS compared to MoSCA. It also improves temporal consistency, geometric accuracy, user score, and motion smoothness. In pose estimation, SirenPose outperforms Monst3R with lower absolute trajectory error as well as reduced translational and rotational relative pose error, highlighting its effectiveness in handling rapid motion, complex dynamics, and physically plausible reconstruction.

Figures (5)

• Figure 1: The framework decomposes the input video sequence into two parallel information streams. The low-frequency stream captures global structure and slow dynamics via methods like CAPE, while the high-frequency stream uses the SIREN network to model rapid motion and fine textures. These features are fused to predict dynamic keypoints, supervised by a dual loss: a positional accuracy loss for precise localization and a geometric consistency loss to maintain structural and temporal coherence.
• Figure 2:
• Figure 3: This table shows the performance changes of three metrics when the SirenPose loss function is added to baseline models DS4D and 4DGS. Light blue indicates baseline performance, while dark blue shows post - addition performance. Evidently, model performance is optimized with SirenPose.
• Figure 4: Temporal trends of ATE, RPEtrans, and RPErot for Robust-CVD, Monst3R, and SirenPose. Compared with the baselines, SirenPose exhibits smoother and more stable curves with fewer abrupt spikes, highlighting its improved stability and consistency in frame-wise reconstruction.
• Figure 5: This bar chart presents the results of three metrics after removing different keypoints.