PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes

TL;DR

PhysTalk addresses the gap between open-vocabulary language interfaces and physically grounded 3D Gaussian animations. It reframes the task as Text-to-Physics Translation, using an LLM to emit executable Genesis code that operates directly on a 3DGS via a convex-hull proxy and a skinning pipeline. The approach is training-free, GPU-accelerated, and interactive, supporting multi-material dynamics and real-time user edits without mesh extraction. Experiments across diverse scenes show improved perceptual alignment and realism over baselines while maintaining interactive runtimes, highlighting its potential to democratize physically plausible 3DGS animation.

Abstract

Realistic visual simulations are omnipresent, yet their creation requires computing time, rendering, and expert animation knowledge. Open-vocabulary visual effects generation from text inputs emerges as a promising solution that can unlock immense creative potential. However, current pipelines lack both physical realism and effective language interfaces, requiring slow offline optimization. In contrast, PhysTalk takes a 3D Gaussian Splatting (3DGS) scene as input and translates arbitrary user prompts into real time, physics based, interactive 4D animations. A large language model (LLM) generates executable code that directly modifies 3DGS parameters through lightweight proxies and particle dynamics. Notably, PhysTalk is the first framework to couple 3DGS directly with a physics simulator without relying on time consuming mesh extraction. While remaining open vocabulary, this design enables interactive 3D Gaussian animation via collision aware, physics based manipulation of arbitrary, multi material objects. Finally, PhysTalk is train-free and computationally lightweight: this makes 4D animation broadly accessible and shifts these workflows from a "render and wait" paradigm toward an interactive dialogue with a modern, physics-informed pipeline.

Figures (8)

• Figure 1: PhysTalk enables a wide range of physical behaviors via Text-to-Physics Translation. The first row shows a vase with elastic and rigid regions reacting differently when dropped: while the vase remains rigid, the impact force is transferred to the elastic flowers, which begin to wobble. The second row illustrates altered object motion under lunar gravity.
• Figure 2: PhysTalk overview.Input. Our model leverages a 3DGS scene, a user prompt, and a set of simulation-grounding documents. Text-to-Physics Translation. The text signals are fed to an LLM, which generates code for a physical simulation of the scene object. Physics Simulation. We run the simulation and record, per frame, particle motions (black dots) and deformations (black arrows). Skinning For each Gaussian (red circle), we select $K$ neighboring particles and their deformation gradients (red points and arrows), then apply inverse-distance blending to obtain the Gaussian’s motion and deformation; this is repeated for all Gaussians. Output. The pipeline outputs a physics-based 4D Gaussian animation conditioned on the object and prompt.
• Figure 3: Qualitative comparison of PhysTalk with existing baseline methods across multiple scenes and prompts. First example highlights PhysTalk's ability to faithfully execute user prompts and generate elastic, deformable materials. Second example shows our pipeline applying a realistic push to the bulldozer. Third example shows PhysTalk's compelling performance at fluid simulation.
• Figure 4: Comparison of lava simulations with different surface-tension parameter values.
• Figure 5: User interaction with an elastic bulldozer. Yellow arrows indicate active user-applied forces, while red, blue, and green arrows denote inactive manipulation handles. The interface allows the object to be pushed or dragged during the simulation while remaining governed by the underlying physics.