PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis

TL;DR

PhysWorld tackles data-efficient learning of deformable-object dynamics by integrating a physics-consistent MPM digital twin with a lightweight, property-aware GNN world model trained on diversified synthetic demonstrations. A VLM-guided constitutive-model selection and global-to-local property optimization yield a physics-faithful twin, while VMP-Gen and $P^3$-Pert generate extensive 4D demonstrations to cover diverse motions and material heterogeneity; a GNN then learns spatially-varying dynamics and is fine-tuned with real data to bridge sim-to-real gaps. Experiments across 22 scenarios show competitive predictive accuracy and a 47× faster inference speed than PhysTwin, enabling real-time planning and interactive simulation. The approach reduces data requirements while preserving physical plausibility, offering a scalable route to deformable-object world models for robotics, VR, and AR applications.

Abstract

Interactive world models that simulate object dynamics are crucial for robotics, VR, and AR. However, it remains a significant challenge to learn physics-consistent dynamics models from limited real-world video data, especially for deformable objects with spatially-varying physical properties. To overcome the challenge of data scarcity, we propose PhysWorld, a novel framework that utilizes a simulator to synthesize physically plausible and diverse demonstrations to learn efficient world models. Specifically, we first construct a physics-consistent digital twin within MPM simulator via constitutive model selection and global-to-local optimization of physical properties. Subsequently, we apply part-aware perturbations to the physical properties and generate various motion patterns for the digital twin, synthesizing extensive and diverse demonstrations. Finally, using these demonstrations, we train a lightweight GNN-based world model that is embedded with physical properties. The real video can be used to further refine the physical properties. PhysWorld achieves accurate and fast future predictions for various deformable objects, and also generalizes well to novel interactions. Experiments show that PhysWorld has competitive performance while enabling inference speeds 47 times faster than the recent state-of-the-art method, i.e., PhysTwin.

Figures (5)

• Figure 1: Overview of PhysWorld. The framework first constructs a physics-consistent digital twin from videos, then uses it to generate diverse 4D demonstrations, and finally trains a GNN-based world model for real-time future state prediction.
• Figure 2: Visual results of action-conditioned future prediction. Our method's predicted positions show closer alignment with ground truth compared to PhysTwin.
• Figure 3: Generalization to unseen interactions. As representative examples, we consider two unseen interaction scenarios: lifting a pushed rope and rotating a lifted sloth. The results show that PhysWorld generates physically plausible predictions, while PhysTwin suffers from artifacts such as fracture-like rope distortions and unnatural foot folding.
• Figure 4: Examples of model-based planning. With MPPI control, the rope and the zebra doll are transferred from the initial configurations to the target ones.
• Figure 5: Additional visual results of action-conditioned future prediction. Our method's predicted positions show closer alignment with ground truth compared to PhysTwin.