Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

TL;DR

This paper tackles the challenge of evaluating whether vision-language navigation agents truly ground visual content. It introduces a Multi-Branch Architecture (MBA) that ingests multiple visual inputs—original RGB, depth, perturbed views, and random noise—and fuses their predictions via learnable branch weights to improve navigation generalization. Empirical results across R2R, REVERIE, and SOON show that visual perturbations, especially when combined across multiple branches, can yield substantial performance gains, surpassing state-of-the-art on several benchmarks. The findings highlight that visual modalities remain underexploited in VLN, and that deliberate perturbations can serve as a robust mechanism to enhance generalization and diagnosis of visual grounding in embodied AI tasks.

Abstract

Autonomous navigation guided by natural language instructions in embodied environments remains a challenge for vision-language navigation (VLN) agents. Although recent advancements in learning diverse and fine-grained visual environmental representations have shown promise, the fragile performance improvements may not conclusively attribute to enhanced visual grounding,a limitation also observed in related vision-language tasks. In this work, we preliminarily investigate whether advanced VLN models genuinely comprehend the visual content of their environments by introducing varying levels of visual perturbations. These perturbations include ground-truth depth images, perturbed views and random noise. Surprisingly, we experimentally find that simple branch expansion, even with noisy visual inputs, paradoxically improves the navigational efficacy. Inspired by these insights, we further present a versatile Multi-Branch Architecture (MBA) designed to delve into the impact of both the branch quantity and visual quality. The proposed MBA extends a base agent into a multi-branch variant, where each branch processes a different visual input. This approach is embarrassingly simple yet agnostic to topology-based VLN agents. Extensive experiments on three VLN benchmarks (R2R, REVERIE, SOON) demonstrate that our method with optimal visual permutations matches or even surpasses state-of-the-art results. The source code is available at here.

Figures (5)

• Figure 1: Overview of our multi-branch architecture with different visual input choices for the VLN agent: original RGB ($v^{OG}$), Perturbed View ($v^{PV}$), corresponding depth images ($v^{D}$) and random noise ($v^{RN}$). Subscripts $g$ and $l$ denote the global and local branches, respectively. SPL metric represents navigational success rate weighted by path length.
• Figure 2: Success Rate (SR) and SPL of the baseline agent with different visual inputs on the REVERIE val-unseen split.
• Figure 3: Overview of the proposed method, encompassing three components: (a) presents the visual input strategies, (b) illustrates the pipeline of the Multi-Branch Architecture (MBA) with the optimal visual input combination (labeled by $\bigstar$ and $\bigstar$), and (c) displays the the single-branch architecture, elucidating the internal structure of local and global branch. $\mathcal{\hat{P}}$ and $\mathcal{\hat{G}}$ represent panorama and topological map with perturbed visual inputs.
• Figure 4: The heatmaps show the impact of each branches with diverse visual inputs on the val-unseen split of the REVERIE, with color intensity reflecting the SPL(%). (a) SPL in DUET for different visual combinations to the base global and local branches. (b) SPL in our MBA model based on DUET for different visual combinations to the ancillary branches.
• Figure 5: Predicted trajectories of MBA and DEUT on R2R Val-unseen split.