Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigation Instruction Generation with BEV Perception and Large Language Models

Sheng Fan, Rui Liu, Wenguan Wang, Yi Yang

TL;DR

Navigation instruction generation benefits from incorporating 3D scene understanding. BEVInstructor fuses Bird's Eye View representations with perspective cues and tunes Multi-Modal LLMs via parameter-efficient prompts, then progressively refines instructions through instance-guided landmarks. The approach achieves state-of-the-art results across indoor and outdoor VLN benchmarks (R2R, REVERIE, UrbanWalk) with notable gains in SPICE and CIDEr and demonstrates strong qualitative grounding of landmarks. These results highlight the importance of 3D geometry and object semantics in language-grounded navigation and point to future directions in multi-sensor fusion and safety-aware instruction generation.

Abstract

Navigation instruction generation, which requires embodied agents to describe the navigation routes, has been of great interest in robotics and human-computer interaction. Existing studies directly map the sequence of 2D perspective observations to route descriptions. Though straightforward, they overlook the geometric information and object semantics of the 3D environment. To address these challenges, we propose BEVInstructor, which incorporates Bird's Eye View (BEV) features into Multi-Modal Large Language Models (MLLMs) for instruction generation. Specifically, BEVInstructor constructs a PerspectiveBEVVisual Encoder for the comprehension of 3D environments through fusing BEV and perspective features. To leverage the powerful language capabilities of MLLMs, the fused representations are used as visual prompts for MLLMs, and perspective-BEV prompt tuning is proposed for parameter-efficient updating. Based on the perspective-BEV prompts, BEVInstructor further adopts an instance-guided iterative refinement pipeline, which improves the instructions in a progressive manner. BEVInstructor achieves impressive performance across diverse datasets (i.e., R2R, REVERIE, and UrbanWalk).

Navigation Instruction Generation with BEV Perception and Large Language Models

TL;DR

Navigation instruction generation benefits from incorporating 3D scene understanding. BEVInstructor fuses Bird's Eye View representations with perspective cues and tunes Multi-Modal LLMs via parameter-efficient prompts, then progressively refines instructions through instance-guided landmarks. The approach achieves state-of-the-art results across indoor and outdoor VLN benchmarks (R2R, REVERIE, UrbanWalk) with notable gains in SPICE and CIDEr and demonstrates strong qualitative grounding of landmarks. These results highlight the importance of 3D geometry and object semantics in language-grounded navigation and point to future directions in multi-sensor fusion and safety-aware instruction generation.

Abstract

Navigation instruction generation, which requires embodied agents to describe the navigation routes, has been of great interest in robotics and human-computer interaction. Existing studies directly map the sequence of 2D perspective observations to route descriptions. Though straightforward, they overlook the geometric information and object semantics of the 3D environment. To address these challenges, we propose BEVInstructor, which incorporates Bird's Eye View (BEV) features into Multi-Modal Large Language Models (MLLMs) for instruction generation. Specifically, BEVInstructor constructs a PerspectiveBEVVisual Encoder for the comprehension of 3D environments through fusing BEV and perspective features. To leverage the powerful language capabilities of MLLMs, the fused representations are used as visual prompts for MLLMs, and perspective-BEV prompt tuning is proposed for parameter-efficient updating. Based on the perspective-BEV prompts, BEVInstructor further adopts an instance-guided iterative refinement pipeline, which improves the instructions in a progressive manner. BEVInstructor achieves impressive performance across diverse datasets (i.e., R2R, REVERIE, and UrbanWalk).
Paper Structure (18 sections, 11 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 18 sections, 11 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Perspective-BEV Visual Encoder
  6. Perspective-BEV Prompt Tuning
  7. Instance-Guided Iterative Refinement
  8. Implementation Details
  9. Experiments
  10. Experimental Settings
  11. Quantitative Results
  12. Qualitative Results
  13. Diagnostic Experiment
  14. Instruction Quality Analysis
  15. Conclusion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: BEVInstructor$_{\!}$ verbalizes$_{\!}$ concise$_{\!}$ navigation$_{\!}$ instructions. Navigation instruction generation is of great value to a wide range of tasks, (i) assisting in navigation for blind individuals, (ii) executing long-term tasks with automatic progress reporting, and (iii) conducting autonomous search and rescue operations in disaster areas.
  • Figure 2: $_{\!}$Overview$_{\!}$ of$_{\!}$BEVInstructor$_{\!}$ for$_{\!}$ navigation$_{\!}$ instruction$_{\!}$ generation. (i) BEV incorporates perspective embeddings by querying for 3D scene understanding (§\ref{['sec:visualencoder']}), (ii) we adopt BEV-Perspective prompt tuning for the cross-modal alignment with MLLMs (§\ref{['sec:alignment']}), (iii) the instructions are generated and improved progressively through multiple refinements (§\ref{['sec:refinement']}). Please refer to §\ref{['sec:methodology']} for more details.
  • Figure 3: Visual comparison results between ground truth and BEVInstructor for instruction generation on REVERIE reverie. See §\ref{['sec:resultfig']} for more details.
  • Figure S1: Comparison results among Ground-Truth, BT-speaker fried2018speaker, EDrop-speaker tan2019learning, CCC-speaker wang2022counterfactual, Lana wang2023lana, and BEVInstructor for instruction generation on R2R anderson2018vision val unseen split. See §\ref{['sec:supplequan']} for more details.