Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimization

TL;DR

This work tackles fine-grained alignment in Vision-and-Language Navigation by introducing FGVLN, a Bayesian Optimization–based adversarial framework that generates fine-grained vision negatives to strengthen cross-modal contrastive learning. The method employs a dual ViLBERT backbone (online and target) with inner-maximization (frame masking via BO) and outer-minimization (training on BO-identified negatives), along with delayed updates to stabilize learning. Empirical results on R2R and REVERIE show improved discriminative and generative VLN performance, including a notable SR and SPL boost in unseen environments and superior embedding quality compared to baselines. The approach provides a practical pathway to leverage informative hard negatives for more precise instruction-path alignment, with ablations validating key design choices and a public code release for reproducibility.

Abstract

This paper addresses the challenge of fine-grained alignment in Vision-and-Language Navigation (VLN) tasks, where robots navigate realistic 3D environments based on natural language instructions. Current approaches use contrastive learning to align language with visual trajectory sequences. Nevertheless, they encounter difficulties with fine-grained vision negatives. To enhance cross-modal embeddings, we introduce a novel Bayesian Optimization-based adversarial optimization framework for creating fine-grained contrastive vision samples. To validate the proposed methodology, we conduct a series of experiments to assess the effectiveness of the enriched embeddings on fine-grained vision negatives. We conduct experiments on two common VLN benchmarks R2R and REVERIE, experiments on the them demonstrate that these embeddings benefit navigation, and can lead to a promising performance enhancement. Our source code and trained models are available at: https://anonymous.4open.science/r/FGVLN.

Figures (4)

• Figure 1: An illustration of existing strategies for generating instruction-based and vision-based path-instruction (PI) pairs, where only coarse-grained negative examples are generated and utilized for vision-based PI samples. $L$ and $V$ denote the instruction and path, $+$ represents the positive samples, while $-$ denotes the negative samples.
• Figure 2: Overview of the proposed Fine-grained VLN (FGVLN). In the Inner Maximization, the Bayesian optimizer evaluates different masks $M$ based on $\mathcal{L}_{PR}^{target}$, this process is repeated several iterations (as denoted by lines in red), and resulting a set of best masks $S_M^*$. In outer minimization procedure, the online model is updated given the FGN batch generated based on $S_M^*$.
• Figure 3: A comparison of the embeddings from the vision encoder trained by different methods.
• Figure 4: An illustration of an example trajectory determined by our framework for a given instruction compared to that determined by Lily. Each robot starts at position $0$ (marked in blue). Our framework selects a path (marked in green) that stops at the top of the stairs, while the baseline selects a path (marked in yellow) that only ascends partway up the stairs before stopping in the middle.