RoboScape: Physics-informed Embodied World Model

TL;DR

RoboScape tackles the gap between visual realism and physical plausibility in embodied world models for robotics by introducing physics-informed priors trained jointly with RGB video generation. It adds temporal depth prediction and adaptive keypoint dynamics to a dual-branch RGB-depth Transformer, enforcing geometry and material-aware motion without external simulators. On a large robotic video dataset, RoboScape achieves state-of-the-art RGB and depth accuracy and demonstrates practical value by improving policies trained on synthetic data and serving as a reliable policy evaluator. The approach offers a scalable, efficient alternative to cascaded physics engines for generating physically plausible robotic data, with implications for safer, more effective real-world deployment.

Abstract

World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. The code is available at: https://github.com/tsinghua-fib-lab/RoboScape.

Figures (12)

• Figure 1: Illustration of the proposed robotic data processing pipeline with physical priors annotation.
• Figure 2: Overview of the physics-informed world model, where physical knowledge is integrated through joint learning of temporal depth estimation and adaptively sampled keypoint dynamics.
• Figure 3: Qualitative results visualization of our model (only the subsequent 8 frames are shown). More results can be found in the appendix.
• Figure 4: Effect of the physics knowledge learning. Omission of temporal depth learning leads to geometric distortions in moving objects, while the absence of key-point dynamics learning results in unreal motion patterns.
• Figure 5: Correlation between the success rate of different world models and the ground-truth simulator. Each point represents a policy, and the trained epochs are shown above the point.