DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments

TL;DR

Discoverse introduces a unified, open-source Real2Sim2Real framework that fuses 3D Gaussian splatting rendering with MuJoCo physics to produce photorealistic, asset-compatible robotic simulations. It supports background and interactive scene assets, a Real2Sim pipeline (scene and object level), and extensive domain randomization to bridge Sim2Real gaps. In experiments on imitation-learning benchmarks, Discoverse achieves state-of-the-art zero-shot transfer compared with MuJoCo, RoboTwin, and SplatSim, with data augmentation further boosting performance and enabling much faster data synthesis. The work lays groundwork for large-scale, end-to-end robotic benchmarks across manipulation, navigation, and multi-agent tasks.

Abstract

We present the first unified, modular, open-source 3DGS-based simulation framework for Real2Sim2Real robot learning. It features a holistic Real2Sim pipeline that synthesizes hyper-realistic geometry and appearance of complex real-world scenarios, paving the way for analyzing and bridging the Sim2Real gap. Powered by Gaussian Splatting and MuJoCo, Discoverse enables massively parallel simulation of multiple sensor modalities and accurate physics, with inclusive supports for existing 3D assets, robot models, and ROS plugins, empowering large-scale robot learning and complex robotic benchmarks. Through extensive experiments on imitation learning, Discoverse demonstrates state-of-the-art zero-shot Sim2Real transfer performance compared to existing simulators. For code and demos: https://air-discoverse.github.io/.

Figures (6)

• Figure 1: Discoverse system overview. Discoverse unifies real-world captures, 3D AIGC, and any existing 3D assets in formats of 3DGS (.ply), mesh (.obj/.stl), and MJCF physical models (.xml), enabling their use as interactive scene nodes (objects and robots) or the background node. We leverage Gaussian splatting as our rendering engine to generate hyper-realistic radiance field rendering of multiple sensor modalities and use MuJoCo as the physical engine to ensure accurate physics. Benefiting from the efficiency and fidelity, Discoverse enables user-definable data generation strategy, evaluation metrics, and algorithms for robotics and embodied AI, empowering a variety of applications, e.g., parallel training, complex robotic benchmarks, etc.
• Figure 2: Discoverse operation flow. We utilize fast tile-based splatting for high-fidelity neural rendering and integrate MuJoCo todorov2012mujoco physical simulator for various robotic utilities.
• Figure 3: Discoverse Real2Sim generation pipeline. We use 3DGS as a universal visual representation and integrate laser scanning, state-of-the-art generative models, and physically-based relighting to boost the geometry and appearance fidelity of the reconstructed radiance fields.
• Figure 4: Visualizations of an AIRBOT Play robotic arm performing three different manipulation tasks in the simulation of Discoverse and in reality.
• Figure 5: Visualizations of an agent exploring a large-scale indoor scene in Discoverse at different timestamps. The yellow boxes indicate the ego-view inputs generated by the Discoverse renderer.