High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

TL;DR

RoboSimGS tackles the data bottleneck in robotic learning by building high-fidelity, physically interactive digital twins from real multi-view scenes. It couples a photorealistic 3D Gaussian Splatting background with mesh-based interactive objects and uses a multi-modal LLM to automatically infer object physics and articulation, enabling zero-shot sim-to-real transfer. The framework is complemented by holistic scene augmentation to generate diverse, robust training data that also enhances state-of-the-art visuomotor policies when combined with real data. Experiments across eight manipulation tasks demonstrate strong zero-shot transfer, data efficiency, and scalable synthesis capability, validating RoboSimGS as a practical bridge between simulation and reality.

Abstract

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.

Figures (6)

• Figure 1: Pipeline of RoboSimGS. Starting from multi-view images, we first perform Scene Reconstruction to create a hybrid representation with a photorealistic 3DGS background and interactive mesh objects. A key step involves using a Multi-modal Large Language Model (MLLM) for automatic Physics Estimation and Articulation Inference. The scene is then aligned with the simulator with Sim2Real Environment Alignment. Finally, we apply Holistic Scene Augmentation to generate diverse simulated data. Policies trained on this data can be deployed directly to the real world.
• Figure 2: Qualitative comparison between simulated data from RoboSimGS (left) and real-world data (right).
• Figure 3: Task illustration. We design eight manipulation tasks for real-world evaluation: Stack Cubes, PickPlace, Deformable PickPlace, Upright Bottle, Move Bottle, Drawer Close, Box Close, Wiping, whose details are shown in Section. \ref{['sec:tasks']}.
• Figure 4: Visualization of policy performance under four challenging generalization settings designed to test robustness.
• Figure 5: Data scaling analysis for Diffusion Policy chi2023diffusion on the Stack Cubes task. The plot compares the success rate of policies trained on varying amounts of real-world data versus purely simulated data generated by RoboSimGS. Notably, the policy trained on 200 simulated demonstrations achieves a success rate comparable to one trained on 100 real-world demonstrations, highlighting the high quality and data efficiency of our generated data.