Embodied Navigation Foundation Model

TL;DR

This work introduces NavFoM, a cross-task and cross-embodiment navigation foundation model trained on over 8 million navigation samples and augmented with open-world QA data. It combines Temporal-Viewpoint Indicator tokens and Budget-Aware Temporal Sampling to unify multimodal inputs from diverse camera setups and horizons, and uses a dual-branch VLM architecture with an LLM-based navigator and a trajectory planner. The model achieves state-of-the-art or competitive results across VLN benchmarks, object search, tracking, and autonomous driving, without task-specific fine-tuning, and demonstrates strong real-world generalization on multiple robot platforms. These findings suggest NavFoM as a foundational step toward generalist, deployable navigation capable of handling varied tasks and embodiments.

Abstract

Navigation is a fundamental capability in embodied AI, representing the intelligence required to perceive and interact within physical environments following language instructions. Despite significant progress in large Vision-Language Models (VLMs), which exhibit remarkable zero-shot performance on general vision-language tasks, their generalization ability in embodied navigation remains largely confined to narrow task settings and embodiment-specific architectures. In this work, we introduce a cross-embodiment and cross-task Navigation Foundation Model (NavFoM), trained on eight million navigation samples that encompass quadrupeds, drones, wheeled robots, and vehicles, and spanning diverse tasks such as vision-and-language navigation, object searching, target tracking, and autonomous driving. NavFoM employs a unified architecture that processes multimodal navigation inputs from varying camera configurations and navigation horizons. To accommodate diverse camera setups and temporal horizons, NavFoM incorporates identifier tokens that embed camera view information of embodiments and the temporal context of tasks. Furthermore, to meet the demands of real-world deployment, NavFoM controls all observation tokens using a dynamically adjusted sampling strategy under a limited token length budget. Extensive evaluations on public benchmarks demonstrate that our model achieves state-of-the-art or highly competitive performance across multiple navigation tasks and embodiments without requiring task-specific fine-tuning. Additional real-world experiments further confirm the strong generalization capability and practical applicability of our approach.

Figures (15)

• Figure 1: We provide an illustration of architecture (left) alongside real-world experiment results (right). The results include cross-task and cross-embodiment performence of our method.
• Figure 2: Benchmark performance of NavFoM, we compare NavFoM with SOTA baselines on each benchmarks. See Sec. \ref{['sec:exp']} for more detials.
• Figure 3: Pipeline of NavFoM. Our method provides a unified framework for handling multiple tasks, including Image QA, Video QA, and Navigation. We organize text tokens and visual tokens using temporal-viewpoint indicator tokens (sec. \ref{['sec:TVI_Tokens']}), as described in Section \ref{['sec:llm_forwarding']}. For question answering, our model employs a conventional language modeling head in an autoregressive manner, while for navigation, it uses a planning head to directly predict trajectories.
• Figure 4: Visualization of Temporal-Viewpoint Indicator (TVI) tokens. We employ a clustering algorithm mcinnes2018umap to map high-dimensional embeddings into a 2D space.
• Figure 5: Visualization of BATS and corresponding time cost. (a) Given a fixed token budget $B = 1600$, we illustrate the sampling probability at different timesteps $t$ for the latest timestep $T$. (b) Given a maximum timestep $T = 125$, we plot the sampling probability across different timesteps $t$ under varying token budgets $B$. (c) We compare the inference time when using BATS versus not using BATS (keeping all frames).