Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

Omkar Sawant, Luca Zanatta, Grzegorz Malczyk, Kostas Alexis

Abstract

This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.

Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

Abstract

This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.
Paper Structure (16 sections, 8 equations, 8 figures, 3 tables)

This paper contains 16 sections, 8 equations, 8 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. PROBLEM FORMULATION
  4. METHODOLOGY
  5. Cross-Modal Wasserstein Autoencoder
  6. Corruption scheme 1 ($S_1$)
  7. Corruption scheme 2 ($S_2$)
  8. Navigation Policy Learning
  9. Reward Function
  10. Network Architecture
  11. Simulation Environment
  12. EVALUATION STUDIES
  13. Evaluations for CMWAE
  14. Evaluations for the RL policy in Aerial Gym
  15. Real-world experiments
  16. ...and 1 more sections

Figures (8)

  • Figure 1: Real-world navigation under depth sensor degradation with preserved grayscale image.
  • Figure 2: Architecture of the Cross-Modal Weighted Autoencoder for joint depth–grayscale representation learning.
  • Figure 3: Depth and grayscale images are encoded into a shared latent space via the cmwae. Later, the cross-modal representation is combined with state information and processed by the rl policy, which outputs the velocity actions for a quadrotor platform.
  • Figure 4: Top-down view of the training environment at each curriculum level.
  • Figure 5: mse and ssim box plots for CMWAE trained under corruption scheme $S_1$
  • ...and 3 more figures