Learning Dynamic Scene Reconstruction with Sinusoidal Geometric Priors

TL;DR

SirenPose tackles dynamic 3D scene reconstruction from monocular video by introducing a loss that fuses SIREN's high-frequency activations with geometric priors on keypoints. The approach formalizes a CAPE-SirenPose objective that captures both global structure and rapid motions, while enforcing spatial-temporal keypoint consistency through a specialized loss and optimization strategy. A large-scale expansion to 600k annotated instances supports robust learning, and experiments on DAVIS demonstrate significant gains in spatiotemporal coherence and geometric accuracy over prior methods. The method advances dynamic scene understanding with potential impact on augmented reality, robotics, and high-fidelity motion analysis in challenging, fast-moving scenes.

Abstract

We propose SirenPose, a novel loss function that combines the periodic activation properties of sinusoidal representation networks with geometric priors derived from keypoint structures to improve the accuracy of dynamic 3D scene reconstruction. Existing approaches often struggle to maintain motion modeling accuracy and spatiotemporal consistency in fast moving and multi target scenes. By introducing physics inspired constraint mechanisms, SirenPose enforces coherent keypoint predictions across both spatial and temporal dimensions. We further expand the training dataset to 600,000 annotated instances to support robust learning. Experimental results demonstrate that models trained with SirenPose achieve significant improvements in spatiotemporal consistency metrics compared to prior methods, showing superior performance in handling rapid motion and complex scene changes.

Figures (4)

• Figure 1: This figure illustrates the overall workflow of the SirenPose framework. At the input stage, the system accepts three different types of data: 3D frames, static images, and video frames. All input data are processed by the “Pose Estimation” module to extract the target's keypoint information. Subsequently, the predicted keypoints are compared with the ground truth keypoints (GT Keypoint). Using the loss function $L_{\text{SirenPose}}$, the framework optimizes the keypoint predictions to ensure accuracy and consistency. This process adapts to various input formats while achieving efficient pose estimation.
• Figure 2: As shown in the figure, our proposed SirenPose architecture consists of the following innovative components: (1) Input video frames are first processed through 3D Gaussian to obtain initial 3D model representations, followed by deformation processing to generate temporally-related 3D model sequences; (2) The core Siren Pose module innovatively combines SIREN's periodic activation characteristics with geometric prior information of keypoints, while simultaneously extracting GT keypoints and predicted keypoints from the input video; (3) Through a carefully designed $L_{SirenPose}$ loss function, this architecture can effectively optimize both spatial localization accuracy and temporal consistency of keypoints, ultimately outputting high-quality and spatio-temporally continuous dynamic 3D models. Experiments demonstrate that this end-to-end architectural design shows significant advantages in handling rapid motion and complex scene changes.
• Figure 3: The figure presents qualitative comparisons between our proposed SirenPose method and two existing approaches, Mosca and DreamScene4D, on a challenging dynamic scene reconstruction task. Each row shows reconstruction results from the respective methods across multiple video frames, demonstrating their ability to handle complex motion and deformation.
• Figure 4: This figure presents a qualitative comparison of pixel-level tracking capabilities across three methods in challenging dynamic scenes. The visualization tracks specific points (indicated by red circles) through multiple frames, demonstrating each method's ability to maintain temporal consistency during complex motions and interactions.