GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

TL;DR

GSWorld presents a closed-loop, photo-realistic robotics manipulation platform that fuses 3D Gaussian Splatting with physics to minimize sim-to-real gaps. It introduces the GSDF asset format to build metric-accurate digital twins from multi-view captures, enabling zero-shot sim2real transfer, DAgger-based continual learning, and reproducible visual benchmarking. The framework supports real-to-sim reconstruction and sim-to-real policy deployment across multiple robots, with tasks designed for zero-shot imitation, continuous policy improvement, and visual RL. By providing standardized assets, fixed camera configurations, and a scalable data-generation pipeline, GSWorld enables apples-to-apples comparisons across embodiments and tasks, accelerating algorithmic progress in robotic manipulation.

Abstract

This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.

Figures (11)

• Figure 1: GSWorld leverages 3DGS reconstruction to render photo-realistic robot scenes, supports multiple policy learning recipes in the simulation, and realizes zero-shot sim2real transfer. GSWorld also applies to policy visual benchmarking and virtual teleoperation for data collection.
• Figure 2: GSWorld provides an interface on top of existing simulators to render photorealistic assets. Our GSDF assets are compatible with existing simulators to use standard formats for rendering visuals ( e.g., depth, segmentation) and computing physics collisions. GSWorld provides a rendering wrapper on top of simulators to make the RGB rendering photo-realistic to support various domain randomization and applications.
• Figure 3: DAgger Data Collection in Simulation. With privileged information provided by simulation, we can record and relive the failure cases during rollouts and generate corrective data for policy adaptation.
• Figure 4: Real World Hardware Platforms. For FR3, we set up two cameras: a third-person camera positioned in front of the robot (front view), and a wrist-mounted camera attached to the robot’s end effector (wrist view). For xArm6, we set up a third-person camera besides the robot (side view) and a wrist view.
• Figure 5: FR3 Task Visualizations. We design 4 real-world robotic tasks that involve distinct manipulation skills and diverse objects on FR3.