SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigation

TL;DR

This work tackles zero-shot vision-and-language navigation by enabling agents to pre-explore an environment and build a Spatial Scene Graph (SSG) that encodes global spatial structure and semantics. SpatialNav uses an agent-centric spatial map, a compass-like visual input, and remote object localization to perform long-horizon planning grounded in the SSG. Empirical results show SpatialNav outperforms existing zero-shot agents across discrete and continuous VLN tasks and narrows the gap to supervised methods, with further gains when using high-quality spatial annotations. The findings demonstrate the value of global spatial representations for generalizable navigation and point toward integrating such representations with learning-based systems for practical robotic deployment.

Abstract

Although learning-based vision-and-language navigation (VLN) agents can learn spatial knowledge implicitly from large-scale training data, zero-shot VLN agents lack this process, relying primarily on local observations for navigation, which leads to inefficient exploration and a significant performance gap. To deal with the problem, we consider a zero-shot VLN setting that agents are allowed to fully explore the environment before task execution. Then, we construct the Spatial Scene Graph (SSG) to explicitly capture global spatial structure and semantics in the explored environment. Based on the SSG, we introduce SpatialNav, a zero-shot VLN agent that integrates an agent-centric spatial map, a compass-aligned visual representation, and a remote object localization strategy for efficient navigation. Comprehensive experiments in both discrete and continuous environments demonstrate that SpatialNav significantly outperforms existing zero-shot agents and clearly narrows the gap with state-of-the-art learning-based methods. Such results highlight the importance of global spatial representations for generalizable navigation.

Figures (4)

• Figure 1: Illustration of local v.s. global perception during navigation. When the instruction mentions a bedroom, agent maybe confused by local perception if multiple bedrooms are present and plausible. In contrast, global spatial information enables the agent to disambiguate these options and make more accurate actions.
• Figure 2: Overview of the spatial scene graph construction. Given the point cloud input, we employ a four-stage annotation pipeline, including floor segmentation via height-based clustering, room segmentation within each floor using geometric heuristics, room classification based on visual observations and object detection from room-level point clouds. The definitions of node entities in the spatial scene graph are illustrated on the right.
• Figure 3: The framework of SpatialNav agent. Based on its current position, SpatialNav queries the SSG to construct an agent-centric spatial map and to retrieve the object semantic descriptions around navigable places. Panoramic visual observations are organized into a single compass-like image. These information, combined with trajectory history and language instructions, constitutes the context of SpatialNav for predicting the next action.
• Figure 4: Comparison between different MLLM backbones. We report the SR, OSR and SPL.