ParticleGS: Learning Neural Gaussian Particle Dynamics from Videos for Prior-free Physical Motion Extrapolation

TL;DR

ParticleGS addresses long-horizon extrapolation of dynamic 3D scenes by learning neural physical dynamics directly from video. It introduces an Encoder–Evolver–Decoder pipeline that represents scenes as Gaussian particles with static properties and shared dynamic fields, evolved by a Neural ODE to predict future motion. A factorized encoding and a physically grounded decoder (using Rodrigues’ rotation) enable high-order, continuous-time dynamics and physically plausible extrapolation without predefined physics priors. Across multiple synthetic and real datasets, ParticleGS achieves state-of-the-art extrapolation while maintaining rendering quality comparable to leading dynamic 3D reconstruction methods, demonstrating effective physics-guided, prior-free dynamic learning.

Abstract

The ability to extrapolate dynamic 3D scenes beyond the observed timeframe is fundamental to advancing physical world understanding and predictive modeling. Existing dynamic 3D reconstruction methods have achieved high-fidelity rendering of temporal interpolation, but typically lack physical consistency in predicting the future. To overcome this issue, we propose ParticleGS, a physics-based framework that reformulates dynamic 3D scenes as physically grounded systems. ParticleGS comprises three key components: 1) an encoder that decomposes the scene into static properties and initial dynamic physical fields; 2) an evolver based on Neural Ordinary Differential Equations (Neural ODEs) that learns continuous-time dynamics for motion extrapolation; and 3) a decoder that reconstructs 3D Gaussians from evolved particle states for rendering. Through this design, ParticleGS integrates physical reasoning into dynamic 3D representations, enabling accurate and consistent prediction of the future. Experiments show that ParticleGS achieves state-of-the-art performance in extrapolation while maintaining rendering quality comparable to leading dynamic 3D reconstruction methods.

Figures (9)

• Figure 1: Example of typical reconstruction method and our method for extrapolation. Given multi-view RGB video observations, our ParticleGS can learn latent physical dynamics and use them to extrapolate future Gaussians with physically consistent motion and appearance. In contrast, existing time-conditioned deformation methods fail to predict plausible future motion.
• Figure 2: Overview of ParticleGS. (a) Existing time-conditioned methods learn a deformation model for each independent discrete time. (b) Our physics-based framework, ParticleGS, uses latent physical states to drive 3D Gaussian deformation, enabling motion extrapolation. (c) The Dynamics Latent Space Encoder maps Gaussians to an initial physical state, factorized into static properties and dynamic fields. (d) The Neural ODEs Evolver learns the underlying physical laws by modeling the continuous-time evolution of the dynamic fields.
• Figure 3: Rendering and visualization of physical states. Motion-similar Gaussians exhibit comparable features, and their physical states change stably over time.
• Figure 4: Qualitative results of extrapolation at a novel view and a future time on the Dynamic Object Dataset (Bat) and the Dynamic Indoor Scene Dataset (Factory). Red boxes indicate ground truth locations, and blue boxes highlight details.
• Figure 6: Toy experiment on extrapolation capabilities. We visualize the predicted trajectories on a 2D damped spiral. Green dots represent observed training data, while red dots indicate the unobserved future. (a) Baseline: The time-conditioned baseline fits the training data well but fails during extrapolation, producing a linear trajectory that deviates from the physical spiral. (b) Neural ODE: By modeling the underlying derivatives, the Neural ODE accurately captures the spiral dynamics, maintaining the correct trajectory even in the unobserved region.