CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMs

TL;DR

This work tackles ObjectNav, where an agent must locate a target category $c$ in unseen environments. CogNav uses an online heterogeneous cognitive map and an LLM-driven finite-state machine with states $BS$, $CS$, $OT$, $CV$, and $TC$ to reason state transitions and select navigation landmarks via prompts. The cognitive map comprises a scene graph, a landmark graph, and a top-down occupancy map, with 3D instance reasoning aided by a vision-language model to enrich semantic and spatial descriptions; a Fast Marching Method guides low-level planning. On HM3D, MP3D, and RoboTHOR, CogNav achieves state-of-the-art results, improving successful navigation rates (e.g., from 62% to 72.5% on HM3D) and demonstrating zero-shot generalization and real-robot feasibility. This work demonstrates that combining LLM reasoning with online cognitive maps can unlock spatial intelligence without explicit policy training.

Abstract

Object goal navigation (ObjectNav) is a fundamental task in embodied AI, requiring an agent to locate a target object in previously unseen environments. This task is particularly challenging because it requires both perceptual and cognitive processes, including object recognition and decision-making. While substantial advancements in perception have been driven by the rapid development of visual foundation models, progress on the cognitive aspect remains constrained, primarily limited to either implicit learning through simulator rollouts or explicit reliance on predefined heuristic rules. Inspired by neuroscientific findings demonstrating that humans maintain and dynamically update fine-grained cognitive states during object search tasks in novel environments, we propose CogNav, a framework designed to mimic this cognitive process using large language models. Specifically, we model the cognitive process using a finite state machine comprising fine-grained cognitive states, ranging from exploration to identification. Transitions between states are determined by a large language model based on a dynamically constructed heterogeneous cognitive map, which contains spatial and semantic information about the scene being explored. Extensive evaluations on the HM3D, MP3D, and RoboTHOR benchmarks demonstrate that our cognitive process modeling significantly improves the success rate of ObjectNav at least by relative 14% over the state-of-the-arts.

Figures (6)

• Figure 1: We propose to model the cognitive process of object goal navigation with an LLM. The LLM is leveraged to reason the transitions of a spectrum of states ranging from exploration to identification via being prompted by a heterogeneous cognitive map. The cognitive map is constructed online and dynamically updated/corrected through, again, prompting the LLM.
• Figure 2: Pipeline of CogNav: Our method takes a posed RGB-D sequence as input and incrementally constructs an online cognitive map, comprising a scene graph, a landmark graph, and an occupancy map. We then perform cognitive map prompting by encoding cognitive information and goal object into a text prompt used to query the LLM to determine the next cognitive state. Based on the state, the LLM is queried again to select a landmark to guide the robot. A deterministic local planner is used to generate a path to the selected landmark.
• Figure 3: Cognitive Map Prompting: We encode knowledge from the cognitive map by constructing landmark-centered prompts encompassing both scene and agent information.
• Figure 4: State-related landmarks (red dots) activated by different cognitive states. See the supplementary for a detailed explanation of landmarks.
• Figure 5: Navigation process visualization of CogNav. We provide visual results of navigation process for one synthetic scene and one real-world one. Cognitive maps encode scene information and facilitate landmark prompting, enabling the agent to explore environments efficiently and identify target objects accurately. More results can be found in the supplementary.