Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mirage2Matter: A Physically Grounded Gaussian World Model from Video

Zhengqing Gao, Ziwen Li, Xin Wang, Jiaxin Huang, Zhenyang Ren, Mingkai Shao, Hanlue Zhang, Tianyu Huang, Yongkang Cheng, Yandong Guo, Runqi Lin, Yuanyuan Wang, Tongliang Liu, Kun Zhang, Mingming Gong

TL;DR

Mirage2Matter addresses the bottleneck of real-world interaction data for embodied AI by constructing a physics-grounded, photorealistic Gaussian world from ordinary multi-view videos. The approach combines two asset streams (scene and object 3D Gaussian representations) with a calibration-and-alignment pipeline to ensure consistent sim-to-real scaling and behavior, enabling scalable data generation for Vision–Language–Action training. Key contributions include a unified Gaussian world with physically grounded geometry, a cross-domain alignment strategy, and a hybrid rendering pipeline that preserves dynamics while delivering realism; experiments show strong zero-shot sim-to-real generalization compared to baselines and narrowing gaps relative to real-data training. This framework offers a practical, scalable path for embodied AI training, potentially accelerating real-world deployment and broadening the use of photorealistic simulations in robotics and VR.

Abstract

The scalability of embodied intelligence is fundamentally constrained by the scarcity of real-world interaction data. While simulation platforms provide a promising alternative, existing approaches often suffer from a substantial visual and physical gap to real environments and rely on expensive sensors, precise robot calibration, or depth measurements, limiting their practicality at scale. We present Simulate Anything, a graphics-driven world modeling and simulation framework that enables efficient generation of high-fidelity embodied training data using only multi-view environment videos and off-the-shelf assets. Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS), seamlessly capturing fine-grained geometry and appearance from video. We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target, enabling accurate scale alignment between the reconstructed scene and the real world. Together, these components provide a unified, editable, and physically grounded world model. Vision Language Action (VLA) models trained on our simulated data achieve strong zero-shot performance on downstream tasks, matching or even surpassing results obtained with real-world data, highlighting the potential of reconstruction-driven world modeling for scalable and practical embodied intelligence training.

Mirage2Matter: A Physically Grounded Gaussian World Model from Video

TL;DR

Mirage2Matter addresses the bottleneck of real-world interaction data for embodied AI by constructing a physics-grounded, photorealistic Gaussian world from ordinary multi-view videos. The approach combines two asset streams (scene and object 3D Gaussian representations) with a calibration-and-alignment pipeline to ensure consistent sim-to-real scaling and behavior, enabling scalable data generation for Vision–Language–Action training. Key contributions include a unified Gaussian world with physically grounded geometry, a cross-domain alignment strategy, and a hybrid rendering pipeline that preserves dynamics while delivering realism; experiments show strong zero-shot sim-to-real generalization compared to baselines and narrowing gaps relative to real-data training. This framework offers a practical, scalable path for embodied AI training, potentially accelerating real-world deployment and broadening the use of photorealistic simulations in robotics and VR.

Abstract

The scalability of embodied intelligence is fundamentally constrained by the scarcity of real-world interaction data. While simulation platforms provide a promising alternative, existing approaches often suffer from a substantial visual and physical gap to real environments and rely on expensive sensors, precise robot calibration, or depth measurements, limiting their practicality at scale. We present Simulate Anything, a graphics-driven world modeling and simulation framework that enables efficient generation of high-fidelity embodied training data using only multi-view environment videos and off-the-shelf assets. Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS), seamlessly capturing fine-grained geometry and appearance from video. We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target, enabling accurate scale alignment between the reconstructed scene and the real world. Together, these components provide a unified, editable, and physically grounded world model. Vision Language Action (VLA) models trained on our simulated data achieve strong zero-shot performance on downstream tasks, matching or even surpassing results obtained with real-world data, highlighting the potential of reconstruction-driven world modeling for scalable and practical embodied intelligence training.
Paper Structure (46 sections, 16 equations, 5 figures, 4 tables)

This paper contains 46 sections, 16 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reconstruction-based World Models
  4. Generative-based World Models
  5. Method
  6. Photorealistic Scene and Object Reconstruction
  7. 3D Gaussian Splatting
  8. Scene reconstruction (photorealistic background).
  9. Object Reconstruction (photorealistic foreground assets).
  10. Mesh Generation (physics-ready geometry).
  11. Cross-domain Alignment
  12. Scene Alignment: SfM $\rightarrow$ Genesis (pre-align before 3DGS training).
  13. Object Alignment: Object 3DGS $\leftrightarrow$ Mesh (post-align).
  14. Applying the Transformation to Object 3DGS Parameters.
  15. Calibration and Sim-to-Real Alignment Strategy
  16. ...and 31 more sections

Figures (5)

  • Figure 1: Overview of the Mirage2Matter framework. We reconstruct photorealistic scenes and objects from multi-view videos using 3DGS, align them to a robot-centric frame, and compose a unified world model for physics-based interaction. Hybrid rendering and motion planning are then used to generate scalable visual data for Vision–Language–Action training.
  • Figure 2: Visualization of scene–robot alignment. Left: unaligned scene point cloud (colored) and robot point cloud (blue). Right: aligned point clouds after scaled ICP between the calibration-board region and the robot base.
  • Figure 3: Object-level alignment between 3DGS and mesh representations. Blue points denote the mesh converted to a point cloud, and red points denote the corresponding 3DGS point cloud. Left: object representations. Right: aligned result after similarity-based ICP.
  • Figure 4: Sim-to-real egocentric camera alignment after calibration. Left: real head-mounted camera image. Right: simulated camera rendering. Under identical arm configurations, the manipulator appears at the same relative position in both views, demonstrating accurate camera alignment.
  • Figure 5: Qualitative comparison of different methods across multiple manipulation tasks. Each row corresponds to a specific simulator, from top to bottom: (1) RoboSimGS, (2) DISCOVERSE, (3) Ours, and (4) Real Robot. Each column represents a distinct task, from left to right: Grasp Banana, Grasp Bread, Push Banana, and Press Button. The images are keyframes extracted from videos recorded during task execution. Note that our method (third row) generates robust policies that successfully accomplish the manipulation goals in every scenario. In contrast to the baselines, our approach exhibits behavior that is both visually realistic and functionally effective, closely mirroring the successful execution of the real robot (bottom row), demonstrating superior fidelity compared to RoboSimGS and DISCOVERSE.