ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstacles

TL;DR

ReaDy-Go tackles the challenge of RGB-only navigation in real-world dynamic environments by closing the sim-to-real gap through a photorealistic real-to-sim dynamic Gaussian Splatting (GS) pipeline. It combines a static GS background with animatable human GS obstacles and motion-generation from 2D trajectories to synthesize photorealistic dynamic scenarios, generating datasets for imitation-learning-based navigation policies. The framework includes a GS-based dynamic simulator, an environment-specific data-generation and planning pipeline, and a lightweight policy network trained on expert actions, achieving robust sim-to-real transfer and demonstrated zero-shot generalization to unseen environments. This approach reduces data collection to a single deployment video per environment and improves safety and efficiency for deployment in households, restaurants, and factories.

Abstract

Visual navigation models often struggle in real-world dynamic environments due to limited robustness to the sim-to-real gap and the difficulty of training policies tailored to target deployment environments (e.g., households, restaurants, and factories). Although real-to-sim navigation simulation using 3D Gaussian Splatting (GS) can mitigate this gap, prior works have assumed only static scenes or unrealistic dynamic obstacles, despite the importance of safe navigation in dynamic environments. To address these issues, we propose ReaDy-Go, a novel real-to-sim simulation pipeline that synthesizes photorealistic dynamic scenarios for target environments. ReaDy-Go generates photorealistic navigation datasets for dynamic environments by combining a reconstructed static GS scene with dynamic human GS obstacles, and trains policies robust to both the sim-to-real gap and moving obstacles. The pipeline consists of three components: (1) a dynamic GS simulator that integrates scene GS with a human animation module, enabling the insertion of animatable human GS avatars and the synthesis of plausible human motions from 2D trajectories, (2) navigation dataset generation for dynamic environments that leverages the simulator, a robot expert planner designed for dynamic GS representations, and a human planner, and (3) policy learning using the generated datasets. ReaDy-Go outperforms baselines across target environments in both simulation and real-world experiments, demonstrating improved navigation performance even after sim-to-real transfer and in the presence of moving obstacles. Moreover, zero-shot sim-to-real deployment in an unseen environment indicates its generalization potential. Project page: https://syeon-yoo.github.io/ready-go-site/.

Figures (5)

• Figure 1: The proposed real-to-sim dynamic environment simulation pipeline for visual navigation. ReaDy-Go generates photorealistic navigation datasets for dynamic scenarios and trains environment-specific visual navigation policies from these datasets. The resulting policies demonstrate robustness to the sim-to-real gap and moving obstacles.
• Figure 2: ReaDy-Go overview. The proposed photorealistic simulation pipeline for visual navigation in dynamic environments consists of three main components: (1) a real-to-sim dynamic 3D Gaussian Splatting (GS) simulator with animatable human GS avatars, (2) photorealistic navigation dataset generation for dynamic scenarios, and (3) visual navigation policy training.
• Figure 3: Visualization of the robot expert planner. (a) The robot follows a collision-free path (red) from start (green) to goal (blue). (b) When a dynamic obstacle (human point cloud in red; inflated region in magenta) makes the path unsafe, the robot follows a replanned path (yellow).
• Figure 4: Qualitative novel-view synthesis results from the proposed dynamic GS simulation pipeline across diverse viewpoints and environments. ReaDy-Go generates photorealistic, geometrically consistent dynamic scenarios with natural human motion from novel viewpoints, enabling navigation dataset generation for target deployment environments.
• Figure 5: Real-world experiments. ReaDy-Go demonstrates robust real-world visual navigation performance after sim-to-real transfer in target environments (a--c) and an unseen environment (d), in both Static and Dynamic.