Decoupling Motion and Geometry in 4D Gaussian Splatting

TL;DR

This work proposes VeGaS, a novel velocity-based 4D Gaussian Splatting framework that decouples Gaussian motion and geometry, and introduces a Galilean shearing matrix that explicitly incorporates time-varying velocity to flexibly model complex non-linear motions.

Abstract

High-fidelity reconstruction of dynamic scenes is an important yet challenging problem. While recent 4D Gaussian Splatting (4DGS) has demonstrated the ability to model temporal dynamics, it couples Gaussian motion and geometric attributes within a single covariance formulation, which limits its expressiveness for complex motions and often leads to visual artifacts. To address this, we propose VeGaS, a novel velocity-based 4D Gaussian Splatting framework that decouples Gaussian motion and geometry. Specifically, we introduce a Galilean shearing matrix that explicitly incorporates time-varying velocity to flexibly model complex non-linear motions, while strictly isolating the effects of Gaussian motion from the geometry-related conditional Gaussian covariance. Furthermore, a Geometric Deformation Network is introduced to refine Gaussian shapes and orientations using spatio-temporal context and velocity cues, enhancing temporal geometric modeling. Extensive experiments on public datasets demonstrate that VeGaS achieves state-of-the-art performance.

Figures (6)

• Figure 1: Graphic illustration of the proposed VeGaS in comparison with 4DGS. (a) Both methods share the same random initialization. (b) During training, VeGaS fits trajectories (as illustrated by the blue curves) using a decoupled motion and geometry formulation, whereas 4DGS adopts a coupled modeling scheme. (c) At inference, VeGaS employs time-varying velocity and geometry to capture complex trajectories and deformations, while 4DGS assumes constant velocity and time-invariant geometry. (d) Qualitative results show that VeGaS yields higher rendering fidelity, whereas 4DGS exhibits noticeable artifacts.
• Figure 2: Overview of our proposed velocity-based decoupling of motion and geometry for 4D Gaussian Splatting. (a) Original velocity $v_0$ is transformed using the time-varying velocity $v(t)$ calculated from the shearing matrix, enabling original 4D Gaussians to move along non-linear trajectories in continuous time. (b) The deformation network predicts geometric transformations of Gaussians at any time based on velocity information, time query $t$, and 4D Gaussian center $\mu_{4D}$. (c) Combining the velocity and geometry transformations, the rendered images are obtained through differentiable rasterization of transformed Gaussians at each frame.
• Figure 3: Qualitative result on the Neural 3D Video dataset. Our method exhibits noticeably higher visual quality compared to others.
• Figure 4: Qualitative results on the D-NeRF dataset, where our method captures finer details than other methods.
• Figure 5: Visualization of ablation studies on the Neu3DV dataset, where we individually remove velocity and geometric modeling to evaluate their impact on the model's performance. Continuous video frames are extracted to observe the effects of removing these components on the model's performance.

Theorems & Definitions (10)

• Definition 3.1: Galilean Shearing
• Remark 3.2
• Theorem 3.3: Schur Complement Invariance
• Definition 3.1: Schur Complement
• Remark 3.2: Probabilistic Interpretation
• Theorem 4.1: Schur Complement Invariance
• proof
• Lemma 6.1: Invariance of SPSD under Congruence Transformation.
• proof
• Corollary 6.2: Preservation of Positive Definiteness