SeqWalker: Sequential-Horizon Vision-and-Language Navigation with Hierarchical Planning

TL;DR

This paper tackles Sequential-Horizon Vision-and-Language Navigation (SH-VLN), where agents must execute multi-task trajectories described by long, structured instructions in large, persistent scenes. It introduces SeqWalker, a hierarchical planner with a High-Level Perception Planner for local instruction segmentation and a Low-Level Action Planner leveraging an Exploration-and-Verification (EaV) strategy, along with a Map-Encoder to maintain scene maps. To evaluate SH-VLN, the authors extend IVLN to SH-IR2R-CE, enriching instructions and stitching trajectories to form long-horizon tasks, and demonstrate that SeqWalker achieves state-of-the-art performance on both SH-VLN and traditional IVLN benchmarks. The work highlights substantial gains from local-instruction focusing, entropy-guided phrase selection, and trajectory correction, with practical implications for robust, long-horizon embodied navigation in dynamic environments.

Abstract

Sequential-Horizon Vision-and-Language Navigation (SH-VLN) presents a challenging scenario where agents should sequentially execute multi-task navigation guided by complex, long-horizon language instructions. Current vision-and-language navigation models exhibit significant performance degradation with such multi-task instructions, as information overload impairs the agent's ability to attend to observationally relevant details. To address this problem, we propose SeqWalker, a navigation model built on a hierarchical planning framework. Our SeqWalker features: i) A High-Level Planner that dynamically selects global instructions into contextually relevant sub-instructions based on the agent's current visual observations, thus reducing cognitive load; ii) A Low-Level Planner incorporating an Exploration-Verification strategy that leverages the inherent logical structure of instructions for trajectory error correction. To evaluate SH-VLN performance, we also extend the IVLN dataset and establish a new benchmark. Extensive experiments are performed to demonstrate the superiority of the proposed SeqWalker.

Figures (8)

• Figure 1: Illustration of the proposed Sequential-Horizon Vision-and-Language Navigation (SH-VLN) task and our SeqWalker model. Different from traditional VLN, SH-VLN requires navigation agents to follow a sequential multi-task trajectory navigation, while users tend to provide complex long instructions, posing greater challenges. Our SeqWalker adopts a Hierarchical Planning strategy, where the High-Level Planner selects sub-tasks and instruction phrases based on the agent's observations, and the Low-Level Planner provides actions for robust navigation using a proposed Exploration and Verification strategy.
• Figure 2: Illustration of the proposed SeqWalker pipeline. At each time step $t$, the SeqWalker agent receives RGB and depth observations to follow user instructions. SeqWalker has a hierarchical planning framework. It comprises a High-Level Perception Planner, which segments sequential-horizon instructions to obtain the most relevant segmented instruction with the current states. And it also comprises a Low-Level Action Planner, which employs an exploration-verification strategy to achieve progress toward destinations or leverage the inherent logical order of instructions to correct navigation trajectories dynamically.
• Figure 3: Statistics of the transformed datasets. The statistics graph contains seven groups of comparison metrics: Number of Sentences, Percentage of Verbs in Instruction, Average Instruction Length, Number of Trajectories, Percentage of Explicit Locations, Average Trajectories Scene, Phrases Word Number, G1 represents unchanged instructions, and G2 represents our transformed enrichment IR2R-CE dataset. Please note that for better visualization, we mark some metrics as "$A/x$", i.e., and the true value is $A \times x$.
• Figure 4: The ablation studies of different verification thresholds on the SH IR2R-CE dataset. For fixed thresholds, it starts at 0.05 and gradually increases to 0.25, with a pace of 0.05. The learnable thresholds $\delta_{0}$ achieve the best performance. Note that for better visualization, we normalize the evaluation metrics and label the original values ($Maximum$, $Minimum$).
• Figure 5: Illustration for the architecture of the Map-Encoder.