Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

TL;DR

Mobility VLA introduces a hierarchical navigation policy for Multimodal Instruction Navigation with Tours (MINT) that fuses long-context vision-language reasoning with an offline topological-graph navigator. A high-level VLM identifies a goal frame from a demonstration tour based on multimodal instructions, while a low-level planner traverses an offline graph to generate timely actions. Experiments in a real 836 m^2 office and a home-like setup show strong end-to-end performance, with the long-context VLM substantially outperforming baselines and the topological graph proving essential for robust action generation. The approach enables intuitive multimodal interaction with robots using simple tour videos, while also highlighting current limitations in exploration and VLM latency. Overall, Mobility VLA advances practical multimodal navigation by leveraging demonstration priors and structured graphs to bridge perception, reasoning, and action.

Abstract

An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s

Figures (8)

• Figure 1: Mobility VLA architecture. The multimodal user instruction and a demonstration tour video of the environment are used by a long-context VLM (high-level policy) to identify the goal frame in the video. The low-level policy then uses the goal frame and an offline generated topological map (from the tour video using structure-from-motion) to compute a robot action at every timestep.
• Figure 2: Experiment setup.
• Figure 3: Qualitative comparison of Mobility VLA and other approaches on a multimodal instruction. The bottom row shows the intermediate output of each approach.
• Figure 4: Top-down view of the COLMAP result of the office environment: 3D point landmarks and reference image poses (blue).
• Figure 5: Localization error: median ATE = $0.056$m.