Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, Xin Eric Wang

TL;DR

This survey analyzes Vision-and-Language Navigation (VLN), where agents follow natural language instructions to navigate in 3D environments. It organizes tasks and benchmarks by communication complexity and objective, surveys datasets and simulators, and then categorizes VLN methods into representation learning, action strategy learning, data-centric learning, and prior exploration. Key contributions include a structured taxonomy of tasks, a hierarchical classification of methods with representative techniques (pretraining, graph reasoning, memory, RL, planning, and data augmentation), and a forward-looking discussion of challenges such as data scarcity and generalization. The paper highlights opportunities in knowledge grounding, multi-agent collaboration, sim-to-real transfer, privacy considerations, and culturally diverse, multilingual VLN settings, underscoring the practical impact of VLN for real-world embodied AI systems.

Abstract

A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks. Vision-and-Language Navigation (VLN) is a fundamental and interdisciplinary research topic towards this goal, and receives increasing attention from natural language processing, computer vision, robotics, and machine learning communities. In this paper, we review contemporary studies in the emerging field of VLN, covering tasks, evaluation metrics, methods, etc. Through structured analysis of current progress and challenges, we highlight the limitations of current VLN and opportunities for future work. This paper serves as a thorough reference for the VLN research community.

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

TL;DR

This survey analyzes Vision-and-Language Navigation (VLN), where agents follow natural language instructions to navigate in 3D environments. It organizes tasks and benchmarks by communication complexity and objective, surveys datasets and simulators, and then categorizes VLN methods into representation learning, action strategy learning, data-centric learning, and prior exploration. Key contributions include a structured taxonomy of tasks, a hierarchical classification of methods with representative techniques (pretraining, graph reasoning, memory, RL, planning, and data augmentation), and a forward-looking discussion of challenges such as data scarcity and generalization. The paper highlights opportunities in knowledge grounding, multi-agent collaboration, sim-to-real transfer, privacy considerations, and culturally diverse, multilingual VLN settings, underscoring the practical impact of VLN for real-world embodied AI systems.

Abstract

A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks. Vision-and-Language Navigation (VLN) is a fundamental and interdisciplinary research topic towards this goal, and receives increasing attention from natural language processing, computer vision, robotics, and machine learning communities. In this paper, we review contemporary studies in the emerging field of VLN, covering tasks, evaluation metrics, methods, etc. Through structured analysis of current progress and challenges, we highlight the limitations of current VLN and opportunities for future work. This paper serves as a thorough reference for the VLN research community.
Paper Structure (29 sections, 2 figures, 4 tables)

This paper contains 29 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Tasks and Datasets
  3. Initial Instruction
  4. Oracle Guidance
  5. Human Dialogue
  6. Evaluation
  7. VLN Methods
  8. Representation Learning
  9. Pretraining
  10. Semantic Understanding
  11. Graph Representation
  12. Memory-augmented Model
  13. Auxiliary Tasks
  14. Action Strategy Learning
  15. Reinforcement Learning
  16. ...and 14 more sections

Figures (2)

  • Figure 1: The agent and oracle discuss the VLN task in natural language. Both observe and interact with the navigable environment to accomplish a task.
  • Figure 2: Categories of VLN methods. Methods may not be mutually exclusive to an individual category.