Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields

TL;DR

This work introduces Neural Gaussian Force Field (NGFF), a physics-grounded video prediction framework that unifies 3D Gaussian perception with an ode-based neural dynamics model to produce interactive, physically plausible 4D videos from multi-view RGB inputs. NGFF comprises a feed-forward 3D Gaussian reconstruction module and a neural-operator dynamics predictor that computes object-centric force fields, which are integrated to generate trajectories and rendered via differentiable Gaussian splatting. A large-scale GSCollision dataset (≈640k videos, ~4 TB) supports training and evaluation across diverse rigid and soft-body interactions, with strong generalization in spatial, temporal, and compositional settings and demonstrable sim-to-real transfer. The approach outperforms state-of-the-art baselines in dynamic prediction and video generation, while achieving two orders of magnitude faster inference than traditional MP-based simulators, highlighting its potential for scalable, physics-aware world modeling. NGFF’s explicit physics representations enable interpretable reasoning and interactive generation, paving the way for robust physical understanding and planning in embodied AI systems.

Abstract

Predicting physical dynamics from raw visual data remains a major challenge in AI. While recent video generation models have achieved impressive visual quality, they still cannot consistently generate physically plausible videos due to a lack of modeling of physical laws. Recent approaches combining 3D Gaussian splatting and physics engines can produce physically plausible videos, but are hindered by high computational costs in both reconstruction and simulation, and often lack robustness in complex real-world scenarios. To address these issues, we introduce Neural Gaussian Force Field (NGFF), an end-to-end neural framework that integrates 3D Gaussian perception with physics-based dynamic modeling to generate interactive, physically realistic 4D videos from multi-view RGB inputs, achieving two orders of magnitude faster than prior Gaussian simulators. To support training, we also present GSCollision, a 4D Gaussian dataset featuring diverse materials, multi-object interactions, and complex scenes, totaling over 640k rendered physical videos (~4 TB). Evaluations on synthetic and real 3D scenarios show NGFF's strong generalization and robustness in physical reasoning, advancing video prediction towards physics-grounded world models.

Figures (28)

• Figure 1: Capabilities of ngff.ngff is a physics-grounded video prediction framework that unifies perception and dynamics to model complex interactions and synthesize 4D videos. Built on Gaussian representations and force fields, it enables novel-view and novel-background synthesis as well as force-prompted interactive generation (\ref{['sec:videogen']}). Moreover, ngff achieves strong spatial and temporal generalization in dynamic prediction (\ref{['sec:dynamics']}) and can be effectively adapted to real-world scenarios (\ref{['sec:realworld']}).
• Figure 2: Overall framework of ngff. Given unposed RGB inputs, our approach first reconstructs the scene into object-aware 3D Gaussians through feed-forward prediction, followed by segmentation and refinement to handle occlusions and noise. The refined Gaussians are encoded into high-dimensional features and processed by a DeepONet-based neural operator to predict object-centric force fields. These force fields are integrated through ode solvers to simulate realistic dynamics, enabling iterative prediction and rendering of future scene states with maintained physical consistency.
• Figure 3: GSCollision dataset. (a) Distribution of 10 representative objects characterized by density and material hardness (Young's modulus, log scale). The parameter space spans from soft, lightweight materials (e.g., cloth, rope, pillow) in the lower-left region to rigid, dense objects (e.g., bowl, phone) in the upper-right, providing comprehensive coverage of everyday material properties. (b) Dataset composition totaling 4.25 TB across 3,200 scenes and 640k videos. The pie chart shows storage distribution among training and test splits, multi-view initial scene captures, and auxiliary data files. (c) Representative frame gallery across evaluation scenarios: training sequences, longer temporal rollouts, compositional generalization, novel viewpoints, and novel backgrounds, demonstrating the diversity of physical interactions and visual contexts in our benchmark.
• Figure 4: Qualitative comparison of dynamic prediction methods. Temporal progression of multi-object scenes demonstrating ngff's superior trajectory prediction compared to baseline approaches. Each row shows predictions from a different method (ngff, gcn, Pointformer, Traditional mpm) across identical initial conditions, with time advancing from left to right. The scenarios feature complex, rigid-soft body interactions, including deformable objects (pillows, ropes) interacting with rigid bodies (balls, containers) under gravitational and contact forces. ngff maintains physically consistent trajectories and realistic deformation patterns throughout extended rollouts, while baseline methods exhibit drift, unrealistic dynamics, or computational instability. Additional dynamic prediction visualizations are provided in \ref{['sec:supp:vis_dynamic']}.
• Figure 5: Interactive generation under external perturbations. Red arrows indicate applied forces. Left: upward force on fallen pillow; Right: leftward force on cloth affecting ball motion. ngff produces physically consistent responses to interventions, while baseline methods (Cosmos, Veo3) generate unrealistic dynamics that violate physical constraints. Baseline prompts: Cosmos—"modify the pillow...to show a significant, sudden external force stretching it upward into the air, with interactions with panda and miku"; Veo3—"modify the clothing...to show a significant, sudden external force stretching it leftward."