DIV-Nav: Open-Vocabulary Spatial Relationships for Multi-Object Navigation

TL;DR

DIV-Nav addresses open-vocabulary, spatially constrained multi-object navigation by leveraging an LVLM to decompose complex natural language queries into basic object targets and proximity relations. The method grounds these targets on a real-time semantic map (OneMap) and constructs a joint similarity map via a min-intersection of individual target maps, balanced with a max term to guide exploration. Navigation is guided by frontier exploration using the combined map, and LVLM-based validation confirms both target presence and adherence to the original spatial constraints. Evaluations on the MultiON benchmark and real-world Spot deployments show systematic improvements in spatial-relation navigation and real-world success rates, validating the benefit of combining LVLM reasoning with semantic grounding. Limitations include challenges with small or occluded objects and occasional LVLM validation failures, suggesting avenues for more robust mapping and validation in future work.

Abstract

Advances in open-vocabulary semantic mapping and object navigation have enabled robots to perform an informed search of their environment for an arbitrary object. However, such zero-shot object navigation is typically designed for simple queries with an object name like "television" or "blue rug". Here, we consider more complex free-text queries with spatial relationships, such as "find the remote on the table" while still leveraging robustness of a semantic map. We present DIV-Nav, a real-time navigation system that efficiently addresses this problem through a series of relaxations: i) Decomposing natural language instructions with complex spatial constraints into simpler object-level queries on a semantic map, ii) computing the Intersection of individual semantic belief maps to identify regions where all objects co-exist, and iii) Validating the discovered objects against the original, complex spatial constrains via a LVLM. We further investigate how to adapt the frontier exploration objectives of online semantic mapping to such spatial search queries to more effectively guide the search process. We validate our system through extensive experiments on the MultiON benchmark and real-world deployment on a Boston Dynamics Spot robot using a Jetson Orin AGX. More details and videos are available at https://anonsub42.github.io/reponame/

Figures (4)

• Figure 1: Overview of the object searching process. From a spatially-constrained navigation query given in natural language, our system: (a) employs an LVLM module to reduce the complex query into simpler spatial proximity queries, (b) projects these targets into a reusable semantic belief map to guide the search towards regions in the physical space where these targets are more likely to co-exist, (c) detects potential targets through open-set object detection, and (d) continuously validates candidate objects against the complex spatial query via visual inference of an LVLM.
• Figure 2: Overview of DIV-Nav. (1) We Decompose spatially-constrained instructions into object-level queries on a semantic map (a-b); (2) Compute Intersections across similarity maps to find navigation regions where objects co-exist (c-d); (3) Validate high-similarity regions by aproaching the region, and via LVLM reasoning to confirm individual objects and their spatial relationships (e).
• Figure 3: Four real-world experiment scenes. We conducted 15 multi-object navigation episodes across these environments.
• Figure 4: Data of a real world experiment: (top) RGB frames, (a) similarity map for query "plant" , (b) similarity map for query "robot" (c) and the resulting map of value intersection scores.