Depth Transfer: Learning to See Like a Simulator for Real-World Drone Navigation

TL;DR

This work tackles the sim-to-real gap in depth-based drone navigation by learning a latent depth representation with a Variational Autoencoder (VAE) and temporal modeling via an LSTM, then aligning the latent spaces across domains with a gradient reversal layer (GRL) discriminator. The method enables zero-shot transfer of policies trained in a simple IsaacGym simulator to both a photo-realistic AvoidBench simulator and real-world environments, by refining the encoder on stereo-depth data from various sources and using a domain-adversarial loss alongside reconstruction and KL terms. Key contributions include a ResNet-based VAE with min-pooling dilation to preserve fine obstacle details, a temporal latent representation for robust policy input, and a domain-adaptation framework that aligns simulated and real depth latent spaces. Experiments across IsaacGym, AvoidBench, and real-world tests show substantial improvements in obstacle avoidance and navigation performance, demonstrating robust, generalizable depth-based navigation across diverse domains.

Abstract

Sim-to-real transfer is a fundamental challenge in robot reinforcement learning. Discrepancies between simulation and reality can significantly impair policy performance, especially if it receives high-dimensional inputs such as dense depth estimates from vision. We propose a novel depth transfer method based on domain adaptation to bridge the visual gap between simulated and real-world depth data. A Variational Autoencoder (VAE) is first trained to encode ground-truth depth images from simulation into a latent space, which serves as input to a reinforcement learning (RL) policy. During deployment, the encoder is refined to align stereo depth images with this latent space, enabling direct policy transfer without fine-tuning. We apply our method to the task of autonomous drone navigation through cluttered environments. Experiments in IsaacGym show that our method nearly doubles the obstacle avoidance success rate when switching from ground-truth to stereo depth input. Furthermore, we demonstrate successful transfer to the photo-realistic simulator AvoidBench using only IsaacGym-generated stereo data, achieving superior performance compared to state-of-the-art baselines. Real-world evaluations in both indoor and outdoor environments confirm the effectiveness of our approach, enabling robust and generalizable depth-based navigation across diverse domains.

Figures (7)

• Figure 1: (a) is the basic framework of the obstacle-free navigation system. When training, the depth images are generated from IsaacGym and the policy is trained in the simulator. During real-world deployment, the depth images are collected from real environments and the policy is evaluated using the refined encoder. (b) illustrates the forest environment for policy evaluation. (c) shows the trajectory from (b), the point clouds here are only for visualization.
• Figure 2: Domain adaptation for depth transfer. The VAE is trained using depth images from IsaacGym. The encoder maps depth images to a latent space, while the decoder reconstructs the depth images. The GRL and discriminator are used to align the latent spaces of simulated and real-world depth images.
• Figure 3: Depth reconstruction with min-pooling dilation. The first column shows the original depth and after dilation. The second and third columns compare the reconstructed depth images from a standard VAE and a VAE with a ResNet architecture.
• Figure 4: t-SNE visualization of latent space alignment. The plot includes ground truth depth images from IsaacGym (orange), stereo depth images from IsaacGym (yellow), stereo depth images from AvoidBench (blue), and real-world depth images (purple).
• Figure 5: Policy evaluation in IsaacGym with different depth inputs. The left column shows evaluation with ground truth depth images, the middle column shows the policy evaluated with stereo depth images without encoder refinement, and the right column shows the policy evaluated with stereo depth images after domain adaptation.