Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models

TL;DR

The paper addresses the challenge of providing robust scene context to large language models for navigation by introducing a text-based Tag Map that explicitly stores thousands of semantic tags observed from multiple viewpoints. Grounding an LLM is achieved by prompting with the tag vocabulary and enabling a dedicated API (localize_tag, region_region_dist, point_region_dist) to perform spatial reasoning over coarse 3D localizations derived from multi-view tag evidence. The Tag Map demonstrates memory efficiency and competitive precision/recall compared to embedding-based open-vocabulary maps, and real-robot experiments validate grounded navigation using an LLM to generate actionable plans. By defining $P2E$ and $E2P$ metrics and leveraging a simple but effective localization pipeline, the approach offers a practical pathway to scalable, language-grounded robotic navigation with explicit, reportable scene context.

Abstract

Large Language Models (LLM) have emerged as a tool for robots to generate task plans using common sense reasoning. For the LLM to generate actionable plans, scene context must be provided, often through a map. Recent works have shifted from explicit maps with fixed semantic classes to implicit open vocabulary maps based on queryable embeddings capable of representing any semantic class. However, embeddings cannot directly report the scene context as they are implicit, requiring further processing for LLM integration. To address this, we propose an explicit text-based map that can represent thousands of semantic classes while easily integrating with LLMs due to their text-based nature by building upon large-scale image recognition models. We study how entities in our map can be localized and show through evaluations that our text-based map localizations perform comparably to those from open vocabulary maps while using two to four orders of magnitude less memory. Real-robot experiments demonstrate the grounding of an LLM with the text-based map to solve user tasks.

Figures (13)

• Figure 1: Our text-based map grounds an LLM to plan actionable navigation goals given the scene context to address user-specified tasks.
• Figure 2: Our text-based map (tag map) stores viewpoints from a scene and the entities (tags) recognized by an image tagging model from these viewpoints. The stored tags are used to create a context prompt for grounding an LLM to generate navigation plans over the scene. Tags in the map can be coarsely localized in space and fed into the LLM for further spatial reasoning.
• Figure 3: Left: Example of a close-up view discarded by the depth filtering step. Such images tend to generate inaccurate tags due to a lack of context. Right: An image crop results in significant differences in the tags recognized by an image tagging model. Using a crop augmented ensemble of tagging models filters out the inconsistent tags in red.
• Figure 4: Illustrative coarse localization example of "table" with two viewpoints. Spaces covered by the viewpoints are voxelized and each voxel is assigned votes based on the number of viewpoints that contain it. Clustering voxels of at least $v$ votes generates proposals at a confidence level of $v$. Non-maximum suppression is applied to remove redundant proposals.
• Figure 5: Example coarse localizations for various objects and locations, along with images where they were recognized. The large vocabulary of image tagging models allows our framework to localize uncommon categories such as "thermostat", "cowboy boots" and "basketball court".