GaussGym: An open-source real-to-sim framework for learning locomotion from pixels

TL;DR

GaussGym tackles the visual sim-to-real gap in locomotion by delivering a fast photorealistic real-to-sim framework that integrates 3D Gaussian Splitting (3DGS) as a drop-in renderer within a vectorized physics engine. It ingests a broad range of data sources, including smartphone scans, open datasets, and video-model outputs, and uses VGGT for camera geometry plus NKSR for meshes to initialize Gaussian splats, enabling rapid, photorealistic rendering synchronized with physics. Rendering is decoupled from the control loop and augmented with motion blur to enhance realism and transfer robustness, supporting large-scale training across thousands of scenes and enabling zero-shot transfer to real stairs. Collectively, GaussGym provides an open, scalable platform for vision-based locomotion and visual navigation, helping to close the visual sim-to-real gap and accelerate visuomotor policy research.

Abstract

We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/.

Figures (11)

• Figure 1: GaussGym constructs photorealistic worlds from various data sources and renders them in a vectorized physics engine, achieving high visual fidelity and throughput.
• Figure 2: Data collection overview: GaussGym ingests data from various data sources and processes them with VGGT wang2025vggt to obtain extrinsics, intrinsics, and point clouds with normals. The former two data products are used to train 3D Gaussian Splats for rendering, while the latter two are used to estimate the scene collision mesh.
• Figure 3: Velocity-tracking policies trained directly from pixels in GaussGym: Photorealistic environments provide diverse training scenes, enabling policies to follow commanded velocities from RGB input. GaussGym supports large-scale training with synchronized RGB and depth at over 100K $\frac{\text{steps}}{\text{s}}$ across 4,096 parallel environments on an RTX4090 rendering at $640\times480$.
• Figure 4: GaussGym ingests a variety of datasets - including video model outputs - to produce photorealistic training environments for robot learning.
• Figure 5: Rendering RGB and Depth: Since depth is a by-product of the Gaussian Splatting rasterization process, GaussGym also renders depth without increasing rendering time.