DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigation

TL;DR

DOPE tackles Vision-and-Language Navigation by addressing two core issues: insufficient extraction of instruction details and lack of cross-modal object relationships. It introduces Text Semantic Extraction (TSE), Text Object Perception-Augmentation (TOPA), and Image Object Perception-Augmentation (IOPA) to enrich language understanding and cross-modal object modeling, built on a DUET backbone with a dynamic fusion strategy. The approach leverages CLIP and LXMERT-based cross-modal encoders to fuse object cues across language and vision, achieving superior results on R2R and REVERIE and demonstrating improved grounding (RGS/RGSPL) and path efficiency. The results suggest DOPE enhances robustness and accuracy in complex VLN scenarios, offering a practical path toward more capable embodied agents.

Abstract

Vision-and-Language Navigation (VLN) is a challenging task where an agent must understand language instructions and navigate unfamiliar environments using visual cues. The agent must accurately locate the target based on visual information from the environment and complete tasks through interaction with the surroundings. Despite significant advancements in this field, two major limitations persist: (1) Many existing methods input complete language instructions directly into multi-layer Transformer networks without fully exploiting the detailed information within the instructions, thereby limiting the agent's language understanding capabilities during task execution; (2) Current approaches often overlook the modeling of object relationships across different modalities, failing to effectively utilize latent clues between objects, which affects the accuracy and robustness of navigation decisions. We propose a Dual Object Perception-Enhancement Network (DOPE) to address these issues to improve navigation performance. First, we design a Text Semantic Extraction (TSE) to extract relatively essential phrases from the text and input them into the Text Object Perception-Augmentation (TOPA) to fully leverage details such as objects and actions within the instructions. Second, we introduce an Image Object Perception-Augmentation (IOPA), which performs additional modeling of object information across different modalities, enabling the model to more effectively utilize latent clues between objects in images and text, enhancing decision-making accuracy. Extensive experiments on the R2R and REVERIE datasets validate the efficacy of the proposed approach.

Figures (6)

• Figure 1: Overview of the proposed Dual Object Perception-Enhancement Network (DOPE), which primarily comprises three key components: Text Semantic Extraction (TSE), Text Object Perception Augmentation (TOPA), and Image Object Perception Augmentation (IOPA).
• Figure 2: (a) Action Word Cloud and (b) Object Word Cloud.
• Figure 3: Illustration of the Object Perception-Enhancement module.
• Figure 4: Illustration of cross-modal object modeling.
• Figure 5: Explanation of the impact of different Dropout rates on the metrics.