FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks

TL;DR

This work tackles the lack of cross-dataset generalization in Vision-and-Language Navigation by introducing FlexVLN, a hierarchical system that pairs an LLM Planner with a robust Instruction Follower. A verification module and a multi-model execution ensemble mitigate hallucinations and improve action accuracy, enabling effective generalization to out-of-domain VLN datasets. Empirical results on REVERIE, SOON, and CVDN-target show FlexVLN surpasses prior methods and approaches target-dataset performance without extra training. The approach also reduces LLM usage and inference cost, suggesting practical applicability for adaptable, real-world embodied agents.

Abstract

The aspiration of the Vision-and-Language Navigation (VLN) task has long been to develop an embodied agent with robust adaptability, capable of seamlessly transferring its navigation capabilities across various tasks. Despite remarkable advancements in recent years, most methods necessitate dataset-specific training, thereby lacking the capability to generalize across diverse datasets encompassing distinct types of instructions. Large language models (LLMs) have demonstrated exceptional reasoning and generalization abilities, exhibiting immense potential in robot action planning. In this paper, we propose FlexVLN, an innovative hierarchical approach to VLN that integrates the fundamental navigation ability of a supervised-learning-based Instruction Follower with the robust generalization ability of the LLM Planner, enabling effective generalization across diverse VLN datasets. Moreover, a verification mechanism and a multi-model integration mechanism are proposed to mitigate potential hallucinations by the LLM Planner and enhance execution accuracy of the Instruction Follower. We take REVERIE, SOON, and CVDN-target as out-of-domain datasets for assessing generalization ability. The generalization performance of FlexVLN surpasses that of all the previous methods to a large extent.

Figures (4)

• Figure 1: Comparison between traditional VLN methods and FlexVLN. Traditional methods are usually trained on specific datasets and suffer low success rates when generalizing to other datasets. In contrast, our FlexVLN demonstrates exceptional generalization performance without necessitating additional training or fine-tuning efforts on the target dataset. An illustrative example is presented in the bottom right, where starting from the green flag, the LLM Planner transforms the out-of-domain instruction into step-by-step guidances (validated for feasibility by the MLLM) for seamless execution by the Instruction Follower, ultimately leading to the destination (red flag). denotes the navigation node and heading. The Instruction Follower employs an ensemble of three models. When their action decisions diverge (as indicated by the purple symbols), a LLM is employed to determine the optimal action based on guidance.
• Figure 2: FlexVLN facilitates the seamless integration of the LLM Planner and the Instruction Follower (IF) through a five-step pipeline. The LLM Planner utilizes the given OOD instruction, navigation history, and environmental observations obtained in Step1 to determine an exploration strategy, generating a fine-grained guidance that aligns with the Instruction Follower's familiarity in Step2. The feasibility of the guidance is then verified in Step3. The Instruction Follower is responsible for executing low-level navigation actions based on this guidance in Step4. At the end of the navigation process, the Object Locator is tasked to locate the target object in Step5.
• Figure 3: An example on the REVERIE dataset illustrates the collaboration between the LLM Planner and Instruction Follower.
• Figure 4: Visualization of error cases.