DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning

TL;DR

DELAN addresses Vision-and-Language Navigation by introducing dual-level pre-fusion alignment via self-supervised cross-modal contrastive learning. It constructs a dual-level instruction (original plus landmark words) and enforces instruction-history and landmark-observation alignments across global and local representations before fusion. The approach demonstrates consistent improvements across R2R, RxR, R4R, and CVDN, highlighting stronger cross-modal reasoning, robust long-horizon navigation, and effective dialog-based navigation. This pre-fusion, dual-level alignment framework offers a general, model-agnostic mechanism to reduce modality gaps and enhance downstream decision-making in VLN systems.

Abstract

Vision-and-Language navigation (VLN) requires an agent to navigate in unseen environment by following natural language instruction. For task completion, the agent needs to align and integrate various navigation modalities, including instruction, observation and navigation history. Existing works primarily concentrate on cross-modal attention at the fusion stage to achieve this objective. Nevertheless, modality features generated by disparate uni-encoders reside in their own spaces, leading to a decline in the quality of cross-modal fusion and decision. To address this problem, we propose a Dual-levEL AligNment (DELAN) framework by cross-modal contrastive learning. This framework is designed to align various navigation-related modalities before fusion, thereby enhancing cross-modal interaction and action decision-making. Specifically, we divide the pre-fusion alignment into dual levels: instruction-history level and landmark-observation level according to their semantic correlations. We also reconstruct a dual-level instruction for adaptation to the dual-level alignment. As the training signals for pre-fusion alignment are extremely limited, self-supervised contrastive learning strategies are employed to enforce the matching between different modalities. Our approach seamlessly integrates with the majority of existing models, resulting in improved navigation performance on various VLN benchmarks, including R2R, R4R, RxR and CVDN.

Figures (5)

• Figure 1: An Example. The VLN agent needs to align the historical trajectory to the corresponding instructions (e.g. "Go past the couch towards…") during navigation. At each time step, it selects one of the candidate viewpoints by evaluating the current observation in different directions and landmarks from the instructions (e.g. "bedroom").
• Figure 2: Comparison of pipelines for traditional VL and VLN tasks. (a) In VL tasks, vision-language pre-fusion alignment is often employed to encourage modality interaction. (b) However in VLN filed, there is a blank in pre-fusion alignment, which is solved by our dual-level alignment framework.
• Figure 3: The pipeline of VLN model enhanced with DELAN framework. Landmark words are extracted from the original instructions, and three unimodal encoders encode four types of tokens: instructions, landmarks, histories, and observations. Then, the alignments are employed at pre-fusion stage to enhance cross-modal interaction and action decision.
• Figure 4: Overview of our DELAN framework. The alignments are employed at both instruction-trajectory and landmark-observation levels. Specifically, we implement instruction-trajectory contrast across global (Instruction, Trajectory) and local (Word, Viewpoint) representations using complete instructions and trajectories. At the landmark-observation level, we perform contrast at each time step.
• Figure 5: Trajectories of DUET and DUET+DELAN in R2R-unseen environments. Yellow, green, and red nodes signify the start, target, and incorrect endpoints respectively. Red rectangles correspond the red pats of the instructions.