Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models

Yu Yan, Rongtao Xu, Jiazhao Zhang, Peiyang Li, Xiaodan Liang, Jianqin Yin

TL;DR

Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments.

Abstract

Recent research on Vision-and-Language Navigation (VLN) indicates that agents suffer from poor generalization in unseen environments due to the lack of realistic training environments and high-quality path-instruction pairs. Most existing methods for constructing realistic navigation scenes have high costs, and the extension of instructions mainly relies on predefined templates or rules, lacking adaptability. To alleviate the issue, we propose InstruGen, a VLN path-instruction pairs generation paradigm. Specifically, we use YouTube house tour videos as realistic navigation scenes and leverage the powerful visual understanding and generation abilities of large multimodal models (LMMs) to automatically generate diverse and high-quality VLN path-instruction pairs. Our method generates navigation instructions with different granularities and achieves fine-grained alignment between instructions and visual observations, which was difficult to achieve with previous methods. Additionally, we design a multi-stage verification mechanism to reduce hallucinations and inconsistency of LMMs. Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments. Code is available at https://github.com/yanyu0526/InstruGen.

InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models

TL;DR

Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments.

Abstract

Recent research on Vision-and-Language Navigation (VLN) indicates that agents suffer from poor generalization in unseen environments due to the lack of realistic training environments and high-quality path-instruction pairs. Most existing methods for constructing realistic navigation scenes have high costs, and the extension of instructions mainly relies on predefined templates or rules, lacking adaptability. To alleviate the issue, we propose InstruGen, a VLN path-instruction pairs generation paradigm. Specifically, we use YouTube house tour videos as realistic navigation scenes and leverage the powerful visual understanding and generation abilities of large multimodal models (LMMs) to automatically generate diverse and high-quality VLN path-instruction pairs. Our method generates navigation instructions with different granularities and achieves fine-grained alignment between instructions and visual observations, which was difficult to achieve with previous methods. Additionally, we design a multi-stage verification mechanism to reduce hallucinations and inconsistency of LMMs. Experimental results demonstrate that agents trained with path-instruction pairs generated by InstruGen achieves state-of-the-art performance on the R2R and RxR benchmarks, particularly in unseen environments. Code is available at https://github.com/yanyu0526/InstruGen.

Paper Structure

This paper contains 14 sections, 2 equations, 6 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Vision-and-Language Navigation
  4. Data-centric VLN approaches
  5. METHOD
  6. Trajectory Generation
  7. Instruction Generation
  8. Trajectory Judgment
  9. EXPERIMENTS
  10. Experimental Setup
  11. Pre-Training with InstruGen Data
  12. Comparison with State-of-the-Arts
  13. Zero/One-Shot Navigation
  14. CONCLUSIONS And Limitations

Figures (6)

  • Figure 1: Generate navigation instructions for trajectories sampled from YouTube videos. Instructions generated using R2R templates often deviate from the actual trajectory. The instructions generated by InstruGen are more precise and express environmental information in detail.
  • Figure 2: InstruGen consists of the following stages: (a) Trajectory Generation, (b) Instruction Generation, and (c) Trajectory Judgment.
  • Figure 3: Prompt for the Consistency module
  • Figure 4: Impact of verification mechanism on instruction quality. Yellow and green circles represent the starting and target positions, respectively.
  • Figure 5: The impact of data quality on loss and accuracy during the pre-training process.
  • ...and 1 more figures