Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts

TL;DR

This work introduces VLN-MP, a framework that augments traditional vision-and-language navigation with multi-modal prompts by inserting landmark images alongside textual instructions. It provides a universal data-generation pipeline using GPT-4 for landmark extraction and grounding, plus a Multi-modal Prompts Fusion (MPF) module that unifies image prompts and language tokens for improved navigation, validated across four VLN benchmarks. The authors develop four VLN-MP datasets (R2R-MP, RxR-MP, REVERIE-MP, CVDN-MP), show robust gains over text-only baselines, and demonstrate the approach’s applicability to pre-explore navigation. Overall, VLN-MP broadens VLN capabilities, enabling more reliable grounding from visual priors and improving real-world applicability where image-guided prompts are available.

Abstract

Current Vision-and-Language Navigation (VLN) tasks mainly employ textual instructions to guide agents. However, being inherently abstract, the same textual instruction can be associated with different visual signals, causing severe ambiguity and limiting the transfer of prior knowledge in the vision domain from the user to the agent. To fill this gap, we propose Vision-and-Language Navigation with Multi-modal Prompts (VLN-MP), a novel task augmenting traditional VLN by integrating both natural language and images in instructions. VLN-MP not only maintains backward compatibility by effectively handling text-only prompts but also consistently shows advantages with different quantities and relevance of visual prompts. Possible forms of visual prompts include both exact and similar object images, providing adaptability and versatility in diverse navigation scenarios. To evaluate VLN-MP under a unified framework, we implement a new benchmark that offers: (1) a training-free pipeline to transform textual instructions into multi-modal forms with landmark images; (2) diverse datasets with multi-modal instructions for different downstream tasks; (3) a novel module designed to process various image prompts for seamless integration with state-of-the-art VLN models. Extensive experiments on four VLN benchmarks (R2R, RxR, REVERIE, CVDN) show that incorporating visual prompts significantly boosts navigation performance. While maintaining efficiency with text-only prompts, VLN-MP enables agents to navigate in the pre-explore setting and outperform text-based models, showing its broader applicability.

Figures (5)

• Figure 1: An example of the effectiveness of multi-modal prompts for VLN. Initially, the agent struggles to identify the "wicker chair" due to unfamiliarity. While detailed textual explanations can increase comprehension challenges, introducing a visual prompt from the web is convenient and simplifies the task significantly. In this case, a photo of a wicker chair provides a clear, direct reference to help the agent accurately identify the target object.
• Figure 2: An example of the R2R instruction and the landmark images in three settings of R2R-MP. The aligned setting (green) provides sufficient highly grounded landmark images. The related setting (yellow) presents similar images associated with the phrase but not necessarily the instruction, like the same image for two carpets. The Terminal setting (blue) includes only an image of the last landmark.
• Figure 3: Four stages of our data generation pipeline for transforming textual instructions into multi-modal forms.
• Figure 4: The architecture of the Multi-modal Prompts Fusion (MPF) module. Texts and images are processed separately and then combined to obtain cross-modal representations. Some symbols are omitted for simplicity.
• Figure 5: The average score and proportion of a better alignment of Marky-mT5 and our RxR-MP towards the landmark phrases.