Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DAP: Domain-aware Prompt Learning for Vision-and-Language Navigation

Ting Liu, Yue Hu, Wansen Wu, Youkai Wang, Kai Xu, Quanjun Yin

TL;DR

Vision-and-language navigation (VLN) suffers from a domain gap when applying web-trained pretrained models to in-domain VLN environments. The paper proposes Domain-aware Prompt Learning (DAP), a model-agnostic framework that injects in-domain knowledge via soft visual prompts into the visual encoder while keeping the backbone frozen. DAP leverages in-domain image-text pairs generated with CLIP, uses prompt tuning to ground object- and scene-level semantics, and demonstrates state-of-the-art performance on R2R and REVERIE. The results indicate that lightweight prompt-based adaptation can effectively bridge dataset domain gaps and enhance VLN performance with low training cost.

Abstract

Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-language models are widely used in VLN. However, most of them are trained on web-crawled general-purpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic domain-aware prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.

DAP: Domain-aware Prompt Learning for Vision-and-Language Navigation

TL;DR

Vision-and-language navigation (VLN) suffers from a domain gap when applying web-trained pretrained models to in-domain VLN environments. The paper proposes Domain-aware Prompt Learning (DAP), a model-agnostic framework that injects in-domain knowledge via soft visual prompts into the visual encoder while keeping the backbone frozen. DAP leverages in-domain image-text pairs generated with CLIP, uses prompt tuning to ground object- and scene-level semantics, and demonstrates state-of-the-art performance on R2R and REVERIE. The results indicate that lightweight prompt-based adaptation can effectively bridge dataset domain gaps and enhance VLN performance with low training cost.

Abstract

Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-language models are widely used in VLN. However, most of them are trained on web-crawled general-purpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic domain-aware prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.
Paper Structure (11 sections, 2 equations, 1 figure, 3 tables)

This paper contains 11 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. VLN Problem Setup
  4. Prompt Engineering
  5. Text Generation with the CLIP Model
  6. Domain-aware with Soft Visual Prompts
  7. Experiment
  8. Experimental Setup
  9. Comparison to State-of-the-Art Methods
  10. Effects of Visual Prompts Based on Different Pretrained Models.
  11. Conclusion

Figures (1)

  • Figure 1: The overview of DAP pipeline for VLN. Soft visual prompts are inserted into the input space of the vision encoder, where only the parameters of soft visual prompts and the MLP head are updated during training. DAP focuses on learning soft visual prompts to enhance the adaptation of backbone models to VLN tasks.The visual-and-language model is used for action prediction after the visual module is injected with in-domain knowledge.