LangMap: A Hierarchical Benchmark for Open-Vocabulary Goal Navigation

TL;DR

The paper addresses the need for language-driven, open-vocabulary goal navigation capable of fine-grained grounding in real-world environments. It introduces HieraNav, a multi-granularity navigation framework, and LangMap, a large-scale HM3D-based benchmark with manually curated region and instance level descriptions across scene, room, region, and instance levels. LangMap delivers high-quality, discriminative annotations and over 18k navigation tasks, surpassing prior benchmarks in discriminative accuracy while using substantially fewer words. Experimental results reveal that memory and large-scale multimodal pretraining improve grounding and planning, but robust reasoning for long-tail, small, context-dependent, and multi-goal scenarios remains difficult, highlighting key directions for future research in language-conditioned embodied navigation.

Abstract

The relationships between objects and language are fundamental to meaningful communication between humans and AI, and to practically useful embodied intelligence. We introduce HieraNav, a multi-granularity, open-vocabulary goal navigation task where agents interpret natural language instructions to reach targets at four semantic levels: scene, room, region, and instance. To this end, we present Language as a Map (LangMap), a large-scale benchmark built on real-world 3D indoor scans with comprehensive human-verified annotations and tasks spanning these levels. LangMap provides region labels, discriminative region descriptions, discriminative instance descriptions covering 414 object categories, and over 18K navigation tasks. Each target features both concise and detailed descriptions, enabling evaluation across different instruction styles. LangMap achieves superior annotation quality, outperforming GOAT-Bench by 23.8% in discriminative accuracy using four times fewer words. Comprehensive evaluations of zero-shot and supervised models on LangMap reveal that richer context and memory improve success, while long-tailed, small, context-dependent, and distant goals, as well as multi-goal completion, remain challenging. HieraNav and LangMap establish a rigorous testbed for advancing language-driven embodied navigation. Project: https://bo-miao.github.io/LangMap

Figures (7)

• Figure 1: HieraNav requires agents to interpret natural language and navigate to goals across four semantic levels: scene, room, region, and instance, where success means reaching a target that satisfies the instruction. Instruction-relevant targets are color-coded.
• Figure 2: Analysis of instance-level instructions in GOAT-Bench.
• Figure 3: HieraNav requires agents to interpret natural language instructions and navigate to scene-, room-, region-, and instance-level targets. LangMap establishes the first large-scale benchmark for rigorous and systematic evaluation, featuring single-goal tasks at different semantic levels and multi-goal tasks across mixed levels. Target descriptions incorporate intrinsic attributes, spatial-relational context, and open-world semantics.
• Figure 4: Distribution of region labels in descending frequency.
• Figure 5: Contrastive region annotation. Annotators are provided with region panoramas, corresponding labeled object views, and 3D scene models to compose one concise and one comprehensive description per region, distinguishing it from other same-category regions via visual comparison. A subsequent cross-check ensures clarity and quality.