4D Gaussian Splatting as a Learned Dynamical System

TL;DR

The paper addresses dynamic scene reconstruction under sparse temporal supervision by reframing 4D Gaussian Splatting as a continuous-time dynamical system. It introduces EvoGS, where Gaussian primitives evolve under a learned neural velocity field and are integrated with an RK4 solver, enabling continuous-time extrapolation and compositional motion without per-frame deformations. Key innovations include 4D spatiotemporal feature encoding, a neural dynamical law, Gaussian waypoints for drift stabilization, and a differentiable rendering objective. Empirically, EvoGS delivers improved motion coherence and temporal consistency over deformation-based baselines while maintaining real-time rendering and offering controllable, local motion editing and robust performance under sparse supervision.

Abstract

We reinterpret 4D Gaussian Splatting as a continuous-time dynamical system, where scene motion arises from integrating a learned neural dynamical field rather than applying per-frame deformations. This formulation, which we call EvoGS, treats the Gaussian representation as an evolving physical system whose state evolves continuously under a learned motion law. This unlocks capabilities absent in deformation-based approaches:(1) sample-efficient learning from sparse temporal supervision by modeling the underlying motion law; (2) temporal extrapolation enabling forward and backward prediction beyond observed time ranges; and (3) compositional dynamics that allow localized dynamics injection for controllable scene synthesis. Experiments on dynamic scene benchmarks show that EvoGS achieves better motion coherence and temporal consistency compared to deformation-field baselines while maintaining real-time rendering

Figures (11)

• Figure 1: EvoGS learns a continuous-time dynamical system that governs the evolution of Gaussian primitives. A neural velocity field $v_\theta$ drives their motion through numerical integration. Unlike discrete deformation-based approaches (Fig. \ref{['fig: motivation']}), EvoGS reconstructs unseen timesteps by following the learned dynamics, enabling continuous-time extrapolation and controllable motion composition.
• Figure 2: Top: Canonical deformation methods assign each timestamp an independent mapping from a shared canonical space to produce a set of per-frame transformations (learn what the scene looks like at each time $t$). Bottom:EvoGS instead learns a continuous velocity field that governs Gaussian evolution through time. Dynamics arise from integrating this field to produce reversible trajectories and coherent motion between arbitrarily spaced timestamps. The swirling field visualization shows how local dynamical structure emerges and how injected motion (blue and red) blends into the learned global flow.
• Figure 3: Overview of EvoGS: Given input frames (blue) with photometric supervision, each Gaussian is embedded using 4D spatiotemporal features and evolved through a learned continuous-time velocity field. A neural dynamical law predicts time derivatives of Gaussian attributes, and an ODE solver integrates these dynamics forward or backward to produce unseen future and past states (red), which arise purely from continuous-time evolution.
• Figure 4: Comparison of EvoGS on reconstruction of unseen dynamic human motion on the Jumping Jacks scene. Compared to HexPlane Cao2023HEXPLANE and 4DGS Wu2024_4DGaussianSplatting, which breakdown for unseen timesteps (e.g., limbs rupturing or blurring)
• Figure 5: Extrapolation on real monocular dynamic scenes. Comparison on Split Cookie, Vrig Chicken, and Espresso sequences, where the model must predict frames beyond the observed time range. We include comparisons to yang2023deformable3dgsPumarola21_DNeRFkplanes_2023 in suppl. for completeness.