MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation

TL;DR

MotionPhysics addresses the challenge of deriving physically plausible material parameters from natural language prompts to drive dynamic 3D simulations. It combines a multimodal language model to initialize material priors within realistic ranges and a Learnable Motion Distillation (LMD) loss to extract pure motion cues from pretrained diffusion models, mitigating appearance and geometry biases. A differentiable Gaussian-splat MLS-MPM simulator evolves the scene and gradient-based optimization refines material parameters to match user-described motion. Across 30+ scenes spanning elastic, plastic, fluids, sands, and heterogeneous materials, MotionPhysics delivers state-of-the-art prompt adherence and physical realism with automatic parameter discovery, enabling accessible text-guided physics-based animation. The work lays groundwork for broader text-driven animation tasks and future automatic parameterization improvements.

Abstract

Accurately simulating existing 3D objects and a wide variety of materials often demands expert knowledge and time-consuming physical parameter tuning to achieve the desired dynamic behavior. We introduce MotionPhysics, an end-to-end differentiable framework that infers plausible physical parameters from a user-provided natural language prompt for a chosen 3D scene of interest, removing the need for guidance from ground-truth trajectories or annotated videos. Our approach first utilizes a multimodal large language model to estimate material parameter values, which are constrained to lie within plausible ranges. We further propose a learnable motion distillation loss that extracts robust motion priors from pretrained video diffusion models while minimizing appearance and geometry inductive biases to guide the simulation. We evaluate MotionPhysics across more than thirty scenarios, including real-world, human-designed, and AI-generated 3D objects, spanning a wide range of materials such as elastic solids, metals, foams, sand, and both Newtonian and non-Newtonian fluids. We demonstrate that MotionPhysics produces visually realistic dynamic simulations guided by natural language, surpassing the state of the art while automatically determining physically plausible parameters. The code and project page are available at: https://wangmiaowei.github.io/MotionPhysics.github.io/.

Figures (18)

• Figure 1: MotionPhysics automatically estimates plausible material parameters to support dynamic 3D simulation of diverse materials and object types. Compared to prior work (e.g., PhysFlow liu2025physflow), it more accurately adheres to the user's input prompt (Left), particularly for AI-generated objects (Top: elastic simulation), human-designed objects (Bottom: water simulation), and real-world scans. Image backgrounds are from li2023pac.
• Figure 2: Overview. MotionPhysics simulates physically consistent dynamics from text-guided input prompts by automatically estimating physical parameters for diverse input scenes, including AI-generated, real-world, and human-designed assets.
• Figure 3: Structure Similarity. Left: the same motion pattern (with PCA‑visualized latent codes in the middle) under varying textures and geometries, before optimization. Right: after applying our $L_{\mathrm{LMD}}$, the dynamics become consistent and remain largely unaffected by both texture and geometry.
• Figure 4: Qualitative Evaluation. We compare our method against several baselines across diverse simulation cases, including human-designed objects (e.g., Toothpaste, Left), real-world scenes (e.g., Jam, Middle), and AI-generated objects (e.g., Alien, Right). Red arrows denote input forces, and red circles highlight key regions of difference. See Suppl. for additional results.
• Figure 5: Ablation of $L_{\mathrm{LMD}}$. The corresponding Overall Consistency (OC $\times 10^{-2} \uparrow$) scores are also provided.