HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) is shifting from rigid, step-by-step instruction following toward open-vocabulary, goal-oriented autonomy. Achieving this transition without exhaustive routing prompts requires agents to leverage structural priors. While prior work often assumes computationally heavy 2D/3D metric maps, we instead exploit a lightweight, text-based osmAG (OpenStreetMap Area Graph), a floorplan-level topological representation that is easy to obtain and maintain. However, global planning over a prior map alone is brittle in real-world deployments, where local connectivity can change (e.g., closed doors or crowded passages), leading to execution-time failures. To address this gap, we propose a hierarchical navigation framework HaltNav that couples the robust global planning of osmAG with the local exploration and instruction-grounding capability of VLN. Our approach features an MLLM-based brain module, which is capable of high-level task grounding and obstruction awareness. Conditioned on osmAG, the brain converts the global route into a sequence of localized execution snippets, providing the VLN executor with prior-grounded, goal-centric sub-instructions. Meanwhile, it detects local anomalies via a mechanism we term Reactive Visual Halting (RVH), which interrupts the local control loop, updates osmAG by invalidating the corresponding topology, and triggers replanning to orchestrate a viable detour. To train this halting capability efficiently, we introduce a data synthesis pipeline that leverages generative models to inject realistic obstacles into otherwise navigable scenes, substantially enriching hard negative samples. Extensive experiments demonstrate that our hierarchical framework outperforms several baseline methods without tedious language instructions, and significantly improves robustness for long-horizon vision-language navigation under environmental changes.

Figures (5)

• Figure 1: Overview of the proposed hierarchical navigation framework. To handle abstract user commands, our system decouples navigation into global semantic planning and local physical execution. (Top Left) The GGTD (Graph-Grounded Task Dispatcher) in the MLLM brain reasons the text-based static prior (osmAG) to formulate an initial route. (Bottom Left) The overarching plan is broken down into atomic instructions for the low-level VLN agent to execute. (Bottom Right) Upon encountering unmapped dynamic anomalies, the RVH (Reactive Visual Halting) mechanism interrupts the execution loop. (Top Right) The brain subsequently updates the internal map to prune the blocked passage and autonomously replans a collision-free detour.
• Figure 2: An example of the osmAG map of the second floor of scene 00862 in HM3D dataset ramakrishnan2021hm3d. The map only contains the room info and the passage (e.g., doorway), shown in red dots, connecting the rooms.
• Figure 3: Qualitative examples of Failure-Injected Counterfactual Synthesis. To bridge the sim-to-real gap and generate infinite diverse anomalies, we leverage generative visual inpainting models to modify RGB observations. The top row shows original Seed images denoting traversable paths. The bottom row displays the resulting Augmented images, where semantic obstacles, such as clutter, crowds, or physical hazards—are realistically synthesized. These serve as high-fidelity negative samples for training the reactive halting capability.
• Figure 4: Examples of our proposed methods in HM3D ramakrishnan2021hm3d dataset. The left example illustrates the reactive halting and re-planning ability of our pipeline, given the unexpected obstacle observed. The right one shows an ordinary task without injecting any obstacles.
• Figure 5: An example of our real world test on Fetch robots. Given the instruction and osmAG, the robot sketches a global path starting in the hallway. However, one of the doors at the end of the hallway on the path is unexpectedly closed. With reactive halting module, the robot identified the situation and replanned via the other doorway on the farther side of the hallway.