Visual Semantic Navigation with Real Robots

TL;DR

This work tackles the challenge of transferring Visual Semantic Navigation (VSN) from simulation to real robots by introducing ROS4VSN, a model-agnostic ROS-based framework that enables easy deployment and benchmarking of VSN models on real hardware. The authors implement two leading VSN approaches, PIRLNav and VLV, within ROS4VSN and adapt them for real-world sensor inputs and actuation. Real-world experiments on two distinct robots reveal a substantial performance gap compared to simulation, with PIRLNav dropping from 65% to 21.11% and VLV from 39% to 29.33% success rates, highlighting the impact of perception components like object detectors. The study demonstrates the practicality of a modular, ROS-based evaluation platform for rapid prototyping and comparative analysis of embodied VSN methods, providing a foundation for improving real-world navigation efficiency and robustness.

Abstract

Visual Semantic Navigation (VSN) is the ability of a robot to learn visual semantic information for navigating in unseen environments. These VSN models are typically tested in those virtual environments where they are trained, mainly using reinforcement learning based approaches. Therefore, we do not yet have an in-depth analysis of how these models would behave in the real world. In this work, we propose a new solution to integrate VSN models into real robots, so that we have true embodied agents. We also release a novel ROS-based framework for VSN, ROS4VSN, so that any VSN-model can be easily deployed in any ROS-compatible robot and tested in a real setting. Our experiments with two different robots, where we have embedded two state-of-the-art VSN agents, confirm that there is a noticeable performance difference of these VSN solutions when tested in real-world and simulation environments. We hope that this research will endeavor to provide a foundation for addressing this consequential issue, with the ultimate aim of advancing the performance and efficiency of embodied agents within authentic real-world scenarios. Code to reproduce all our experiments can be found at https://github.com/gramuah/ros4vsn.

Figures (12)

• Figure 1: ObjectNav task is a complex navigation problem. An agent needs to employ vision-based sensors to navigate from a random starting point to specific object goals within the scene. Many different hardware and software components need to be fully integrated to solve it, making it difficult to deploy and test these visual semantic navigation (VSN) models in real robots. Therefore, the best current solutions are trained and tested in virtual environments. Our goal is to bridge the gap between virtual and physical environments by providing a ROS-based framework that simplifies testing and comparing various VSN models on real robotic platforms.
• Figure 2: Architecture scheme of the ROS44VSN framework. It shows the different packages, topics and connections within them and the hardware devices.
• Figure 3: Communications between visual_semantic_navigation and discrete_move packages.
• Figure 4: Acceleration and braking control scheme.
• Figure 5: VSN models integrated into our VSN-ROS.