Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

TL;DR

The paper tackles data scarcity in vision-and-language navigation (VLN) by introducing an adversarial path sampler (APS) that engines counterfactual thinking into data augmentation. APS actively samples challenging navigation paths and uses a back-translated speaker to create instruction-path pairs, optimizing the navigation model through adversarial training with the objective $\mathcal{L}_{APS} = - \mathbb{E}_{p(\mathbf{P}; \pi_{APS})}[\mathcal{L}_{NAV}]$ and policy-gradient updates $\nabla_{\pi_{APS}} \mathcal{L}_{APS} \approx -\sum_{t=1}^T \nabla_{\pi_{APS}} \log p(a_t|a_{1:t-1}; \pi_{APS}) (R-b)$. The approach is model-agnostic and improves performance across multiple VLN baselines (Seq2Seq, Speaker-Follower, RCM) on the Room-to-Room (R2R) dataset in both seen and unseen environments, with additional gains from environment-based pre-exploration that enhances generalization. The results demonstrate that targeting informative counterfactuals via adversarial augmentation yields more robust navigation policies than random augmentation, offering a practical path toward data-efficient VLN and broader counterfactual reasoning applications.

Abstract

Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal by grounding natural language instructions to the visual surroundings. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. In this paper, we explore the use of counterfactual thinking as a human-inspired data augmentation method that results in robust models. Counterfactual thinking is a concept that describes the human propensity to create possible alternatives to life events that have already occurred. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data. In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance. APS also serves to do pre-exploration of unseen environments to strengthen the model's ability to generalize. We evaluate the influence of APS on the performance of different VLN baseline models using the room-to-room dataset (R2R). The results show that the adversarial training process with our proposed APS benefits VLN models under both seen and unseen environments. And the pre-exploration process can further gain additional improvements under unseen environments.

Figures (9)

• Figure 1: The comparison between randomly-sampled (rand) and APS-sampled (aps) under validation-seen set for Seq2Seq over different ratios of augmented path used.
• Figure 2: The learning framework of our adversarial path sampler (APS), where Speaker denotes the back-translated speaker model and NAV denotes the navigation model.
• Figure 3: The architecture of the adversarial path sampler (APS).
• Figure 4: The optimization flow of environment-based pre-exploration under unseen environments. APS samples paths from the unseen environment to optimize NAV and make it more adaptive. Then, NAV runs each instruction in a single turn.
• Figure 5: The comparison between randomly-sampled and APS-sampled under validation-seen and validation-unseen sets for Seq2Seq and Speaker-Follower over different ratios of augmented path used.