Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation

Luo Ling, Bai Qianqian

TL;DR

This work tackles the gap in real-world embodied Vision-and-Language Navigation (VLN) where agents suffer spatial hallucinations when transferring from simulation. It introduces BrainNav, a bio-inspired hierarchical system with five modules—hippocampal memory, visual cortex perception, parietal space builder, prefrontal decision center, and cerebellar motion execution unit—and a dual-map dual-orientation framework to maintain robust spatial cognition in continuous environments. The method achieves zero-shot real-world success on a Limo Pro robot without fine-tuning and outperforms state-of-the-art VLN-CE baselines, validating its effectiveness and practicality. By integrating biological spatial cognition principles with online perception and planning, BrainNav enhances adaptability and reduces spatial hallucinations in dynamic indoor settings, offering a path toward more reliable real-world embodied navigation.

Abstract

Enhancing the spatial perception capabilities of mobile robots is crucial for achieving embodied Vision-and-Language Navigation (VLN). Although significant progress has been made in simulated environments, directly transferring these capabilities to real-world scenarios often results in severe hallucination phenomena, causing robots to lose effective spatial awareness. To address this issue, we propose BrainNav, a bio-inspired spatial cognitive navigation framework inspired by biological spatial cognition theories and cognitive map theory. BrainNav integrates dual-map (coordinate map and topological map) and dual-orientation (relative orientation and absolute orientation) strategies, enabling real-time navigation through dynamic scene capture and path planning. Its five core modules-Hippocampal Memory Hub, Visual Cortex Perception Engine, Parietal Spatial Constructor, Prefrontal Decision Center, and Cerebellar Motion Execution Unit-mimic biological cognitive functions to reduce spatial hallucinations and enhance adaptability. Validated in a zero-shot real-world lab environment using the Limo Pro robot, BrainNav, compatible with GPT-4, outperforms existing State-of-the-Art (SOTA) Vision-and-Language Navigation in Continuous Environments (VLN-CE) methods without fine-tuning.

Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation

TL;DR

This work tackles the gap in real-world embodied Vision-and-Language Navigation (VLN) where agents suffer spatial hallucinations when transferring from simulation. It introduces BrainNav, a bio-inspired hierarchical system with five modules—hippocampal memory, visual cortex perception, parietal space builder, prefrontal decision center, and cerebellar motion execution unit—and a dual-map dual-orientation framework to maintain robust spatial cognition in continuous environments. The method achieves zero-shot real-world success on a Limo Pro robot without fine-tuning and outperforms state-of-the-art VLN-CE baselines, validating its effectiveness and practicality. By integrating biological spatial cognition principles with online perception and planning, BrainNav enhances adaptability and reduces spatial hallucinations in dynamic indoor settings, offering a path toward more reliable real-world embodied navigation.

Abstract

Enhancing the spatial perception capabilities of mobile robots is crucial for achieving embodied Vision-and-Language Navigation (VLN). Although significant progress has been made in simulated environments, directly transferring these capabilities to real-world scenarios often results in severe hallucination phenomena, causing robots to lose effective spatial awareness. To address this issue, we propose BrainNav, a bio-inspired spatial cognitive navigation framework inspired by biological spatial cognition theories and cognitive map theory. BrainNav integrates dual-map (coordinate map and topological map) and dual-orientation (relative orientation and absolute orientation) strategies, enabling real-time navigation through dynamic scene capture and path planning. Its five core modules-Hippocampal Memory Hub, Visual Cortex Perception Engine, Parietal Spatial Constructor, Prefrontal Decision Center, and Cerebellar Motion Execution Unit-mimic biological cognitive functions to reduce spatial hallucinations and enhance adaptability. Validated in a zero-shot real-world lab environment using the Limo Pro robot, BrainNav, compatible with GPT-4, outperforms existing State-of-the-Art (SOTA) Vision-and-Language Navigation in Continuous Environments (VLN-CE) methods without fine-tuning.

Paper Structure

This paper contains 30 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-and-language Navigation
  4. Visual Prompt Engineering
  5. Navigation with Online Mapping
  6. Methodology
  7. Hippocampal Memory Center
  8. Path Backtracking and Historical Experience Reuse
  9. Memory Recall in Decision-Making
  10. Visual Cortex Perception Engine
  11. Multi-Layer Decision Mechanism
  12. Scene Image Processing and Multimodal Information Extraction
  13. Parietal Spatial Builder
  14. Construction and Update of the Coordinate Map
  15. Construction and Update of the Topological Map
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Demonstration of BrainNav's agent responses in the real world. According to human instructions, NaVid, using only the current RGB as input, outputs language actions for the robot to execute.
  • Figure 2: The Overall Architecture of BrainNav
  • Figure 3: Real-world scenes.
  • Figure 4: A successful case of complex instructions in real-world exploration. We provide both the robot's perspective and a third-person perspective, demonstrating its effective multi-task navigation capabilities.