SEMNAV: Enhancing Visual Semantic Navigation in Robotics through Semantic Segmentation

TL;DR

SemNav tackles Visual Semantic Navigation by making semantic segmentation the primary visual input, reducing domain gap between simulated and real-world environments. The approach leverages a newly released SemNav dataset with two segmentation sensors (1630 and 40 categories) integrated into Habitat HM3D, and an imitation-learning policy using a ResNet-50 encoder, GRU memory, and an MLP to map semantic observations and proprioceptive cues to discrete actions. Across extensive Habitat 2.0 experiments and real-world TurtleBot 2 deployments, SemNav demonstrates superior performance to state-of-the-art RGB-based VSN methods, with 40-category segmentation offering robustness and reduced annotation noise; RL fine-tuning further boosts success rates. The work also analyzes segmentation quality, resource demands, and real-world limitations (e.g., stairs, segmentation noise), highlighting practical implications for deploying semantic-aware navigation in real robots and setting directions for future improvements.

Abstract

Visual Semantic Navigation (VSN) is a fundamental problem in robotics, where an agent must navigate toward a target object in an unknown environment, mainly using visual information. Most state-of-the-art VSN models are trained in simulation environments, where rendered scenes of the real world are used, at best. These approaches typically rely on raw RGB data from the virtual scenes, which limits their ability to generalize to real-world environments due to domain adaptation issues. To tackle this problem, in this work, we propose SEMNAV, a novel approach that leverages semantic segmentation as the main visual input representation of the environment to enhance the agent's perception and decision-making capabilities. By explicitly incorporating this type of high-level semantic information, our model learns robust navigation policies that improve generalization across unseen environments, both in simulated and real world settings. We also introduce the SEMNAV dataset, a newly curated dataset designed for training semantic segmentation-aware navigation models like SEMNAV. Our approach is evaluated extensively in both simulated environments and with real-world robotic platforms. Experimental results demonstrate that SEMNAV outperforms existing state-of-the-art VSN models, achieving higher success rates in the Habitat 2.0 simulation environment, using the HM3D dataset. Furthermore, our real-world experiments highlight the effectiveness of semantic segmentation in mitigating the sim-to-real gap, making our model a promising solution for practical VSN-based robotic applications. The code and datasets are accessible at https://github.com/gramuah/semnav

Figures (10)

• Figure 1: Traditional VSN models are typically trained in simulated environments, relying primarily on RGB images. However, the substantial domain gap between simulated and real-world imagery often leads to poor performance when these systems are deployed on real robots. To overcome this limitation, we introduce the SemNav paradigm, where VSN systems use semantic segmentation maps of the environment as their main sensory input. This approach significantly reduces the domain gap between training simulations and real-world scenarios, enabling models to generalize more effectively and perform reliably when tested outside of simulation.
• Figure 2: Comparison between the HM3D dataset and the SemNav 1630 and SemNav 40 novel datasets. In HM3D, objects like chairs are inconsistently labeled with different colors across and within scenes. In contrast, SemNav 1630 assigns a uniform color to objects of the same category while reflecting finer distinctions, such as differentiating kitchen and dining tables. SemNav 40 further simplifies the labeling by merging less critical categories, grouping both types of tables under a shared label to streamline navigation tasks. Figure is best viewed in colour.
• Figure 3: Our proposed architecture for the SemNav model, where the main visual input is a semantic segmentation.
• Figure 4: Top-down views of the houses where ObjectNav real-world experiments were conducted for five object categories.
• Figure 5: Qualitative results in the simulation environment for the different noisy semantic segmentation sensors. (a) RGB sensor output overlaid on the ground truth, (b) semantic segmentation sensor with no noise, (c) semantic segmentation missclassification noise with 10% error, (d) spatially localized perturbations noise error, (e) Border Error noise, (f) ESANet output, (g) semantic segmentation missclassification noise, spatially localized perturbation noise error and border error noise combined.