Vision-and-Language Navigation via Causal Learning

TL;DR

Vision-and-Language Navigation faces pervasive dataset bias that harms generalization. The authors introduce GOAT, a generalized cross-modal causal transformer that integrates back-door adjustment (BACL), front-door adjustment (FACL), and a cross-modal feature pooling (CFP) to deconfound observable and unobservable factors across vision, language, and history using the do-operator $P( ext{Y}|do( ext{X}))$. The CFP module, guided by contrastive learning, builds global confounder dictionaries and aligns multi-modal representations, while BACL and FACL operationalize causal interventions during both pre-training and fine-tuning. Across R2R, RxR, REVERIE, and SOON, GOAT achieves state-of-the-art results, demonstrating improved generalization to unseen environments and more faithful instruction following, with a practical causal-learning pipeline that can be adapted to other embodied AI tasks.

Abstract

In the pursuit of robust and generalizable environment perception and language understanding, the ubiquitous challenge of dataset bias continues to plague vision-and-language navigation (VLN) agents, hindering their performance in unseen environments. This paper introduces the generalized cross-modal causal transformer (GOAT), a pioneering solution rooted in the paradigm of causal inference. By delving into both observable and unobservable confounders within vision, language, and history, we propose the back-door and front-door adjustment causal learning (BACL and FACL) modules to promote unbiased learning by comprehensively mitigating potential spurious correlations. Additionally, to capture global confounder features, we propose a cross-modal feature pooling (CFP) module supervised by contrastive learning, which is also shown to be effective in improving cross-modal representations during pre-training. Extensive experiments across multiple VLN datasets (R2R, REVERIE, RxR, and SOON) underscore the superiority of our proposed method over previous state-of-the-art approaches. Code is available at https://github.com/CrystalSixone/VLN-GOAT.

Figures (17)

• Figure 1: In response to language instructions, the VLN agent is required to navigate to the target based on visual cues. This paper introduces a causal learning pipeline using the do-operator to reduce bias from confounders in VLN action prediction.
• Figure 2: Illustration of the structural causal model of VLN.
• Figure 3: Framework of GOAT, built on the foundation of the dual-scale graph transformer chen2022think. Back-door and front-door adjustment causal learning mechanisms are used for mitigating spurious correlations, enabling unbiased feature learning and decision-making.
• Figure 4: Illustration of BACL and FACL.
• Figure 5: Illustration of the cross-modal feature pooling (CFP).