Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Map-based Generation of Navigation Instructions

Chengzu Li, Chao Zhang, Simone Teufel, Rama Sanand Doddipatla, Svetlana Stoyanchev

TL;DR

This paper tackles navigation instruction generation (VL-GEN) by reframing it as image captioning on a top-down semantic map, instead of using sequences of panorama images. It introduces a semantic-map-augmented dataset built on Room-to-Room with Habitat-derived maps, and a BLIP-based model that fuses TD maps with region, action, and optional panorama inputs, trained with contrastive alignment and prompting. Automatic SPICE scores and human judgments show that incorporating region/action and prompts improves instruction quality, with panoramas offering marginal gains; contrastive loss has mixed impact. Limitations include incomplete region-name encoding and lack of object properties, motivating future work on multi-layer semantic maps and richer annotations. The work provides code and data for the community to advance lightweight, interpretable spatial reasoning in VL-GEN.

Abstract

We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new approach to navigation instruction generation by framing the problem as an image captioning task using semantic maps as visual input. Conventional approaches employ a sequence of panorama images to generate navigation instructions. Semantic maps abstract away from visual details and fuse the information in multiple panorama images into a single top-down representation, thereby reducing computational complexity to process the input. We present a benchmark dataset for instruction generation using semantic maps, propose an initial model and ask human subjects to manually assess the quality of generated instructions. Our initial investigations show promise in using semantic maps for instruction generation instead of a sequence of panorama images, but there is vast scope for improvement. We release the code for data preparation and model training at https://github.com/chengzu-li/VLGen.

Semantic Map-based Generation of Navigation Instructions

TL;DR

This paper tackles navigation instruction generation (VL-GEN) by reframing it as image captioning on a top-down semantic map, instead of using sequences of panorama images. It introduces a semantic-map-augmented dataset built on Room-to-Room with Habitat-derived maps, and a BLIP-based model that fuses TD maps with region, action, and optional panorama inputs, trained with contrastive alignment and prompting. Automatic SPICE scores and human judgments show that incorporating region/action and prompts improves instruction quality, with panoramas offering marginal gains; contrastive loss has mixed impact. Limitations include incomplete region-name encoding and lack of object properties, motivating future work on multi-layer semantic maps and richer annotations. The work provides code and data for the community to advance lightweight, interpretable spatial reasoning in VL-GEN.

Abstract

We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new approach to navigation instruction generation by framing the problem as an image captioning task using semantic maps as visual input. Conventional approaches employ a sequence of panorama images to generate navigation instructions. Semantic maps abstract away from visual details and fuse the information in multiple panorama images into a single top-down representation, thereby reducing computational complexity to process the input. We present a benchmark dataset for instruction generation using semantic maps, propose an initial model and ask human subjects to manually assess the quality of generated instructions. Our initial investigations show promise in using semantic maps for instruction generation instead of a sequence of panorama images, but there is vast scope for improvement. We release the code for data preparation and model training at https://github.com/chengzu-li/VLGen.
Paper Structure (32 sections, 4 equations, 3 figures, 5 tables)

This paper contains 32 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Task Definition and Data
  3. Method
  4. Model Input
  5. Top-down semantic map (TD)
  6. Regions (Reg) and actions (Act)
  7. Panoramic images (Pano)
  8. Model Augmentation
  9. Multimodal alignment with contrastive loss
  10. Augmentation and grounding with prompt
  11. Experiments
  12. Experimental setup
  13. Human Participants and Procedure
  14. Automatic Evaluation Metrics
  15. Evaluation Results
  16. ...and 17 more sections

Figures (3)

  • Figure 1: An example navigation scenario from our new dataset for instruction generation, with the navigation path overlayed on the semantic map.
  • Figure 2: Illustration of the overall model architecture. Text input is encoded with pretrained BLIP text encoder and LSTM, and image input is encoded with the pretrained BLIP encoder. Modules shown in the same color share the weights. The weights of the panorama encoder are fixed.
  • Figure 3: Screenshot of the evaluation interface for human evaluation.