NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance

TL;DR

NavDP presents an end-to-end diffusion-based navigation policy trained exclusively in simulation that transfers zero-shot to real robots with different embodiments. It leverages privileged simulation information (global planning guidance and ESDF-based safety cues) and a critic to select safe trajectories, supported by a transformer-based perception backbone. A large-scale, efficiently generated simulation dataset (56k trajectories, 360+ km across 1,200 scenes) enables robust cross-embodiment generalization, and a preliminary real-to-sim augmentation via Gaussian Splatting further bridges the sim-to-real gap. Experimental results show state-of-the-art performance across multiple robots and environments, with real-to-sim data providing notable gains while preserving generalization.

Abstract

Learning navigation in dynamic open-world environments is an important yet challenging skill for robots. Most previous methods rely on precise localization and mapping or learn from expensive real-world demonstrations. In this paper, we propose the Navigation Diffusion Policy (NavDP), an end-to-end framework trained solely in simulation and can zero-shot transfer to different embodiments in diverse real-world environments. The key ingredient of NavDP's network is the combination of diffusion-based trajectory generation and a critic function for trajectory selection, which are conditioned on only local observation tokens encoded from a shared policy transformer. Given the privileged information of the global environment in simulation, we scale up the demonstrations of good quality to train the diffusion policy and formulate the critic value function targets with contrastive negative samples. Our demonstration generation approach achieves about 2,500 trajectories/GPU per day, 20$\times$ more efficient than real-world data collection, and results in a large-scale navigation dataset with 363.2km trajectories across 1244 scenes. Trained with this simulation dataset, NavDP achieves state-of-the-art performance and consistently outstanding generalization capability on quadruped, wheeled, and humanoid robots in diverse indoor and outdoor environments. In addition, we present a preliminary attempt at using Gaussian Splatting to make in-domain real-to-sim fine-tuning to further bridge the sim-to-real gap. Experiments show that adding such real-to-sim data can improve the success rate by 30\% without hurting its generalization capability.

Figures (6)

• Figure 1: NavDP is solely trained with simulation trajectories but can achieve zero-shot sim-to-real transfer to different types of robots. By learning from the prioritized knowledge in the simulation data, NavDP adaptively selects a safe navigation routes towards the goal without any maps.
• Figure 2: NavDP processes a single RGB-D observation frame along with a navigation goal. The inputs are tokenized and processed through a unified transformer architecture to generate navigation trajectories or evaluate corresponding trajectory values. A safe trajectory is then selected based on these values for execution by the robot.
• Figure 3: Trajectory visualization of on different robots. We project the predicted trajectories back to the image space and colorize them according to the corresponding critic values. The bluer trajectories indicate higher risk, whereas the redder trajectories represent safer paths.
• Figure 4: Ablation results for the NavDP. The left figure illustrate the entire NavDP network can benefit from critic function from test-time selection and training objectives. The middle illustrate the influence of using different tasks for training. The right illustrates the policy performance on both real-to-sim scenes and real-world scenes with respect to different data proportion.
• Figure 5: Examples of our simulation navigation dataset. Our dataset generation pipeline supports texture randomization, view randomization, light randomization and provide photorealistic rendering with BlenderProc.