Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environments

TL;DR

This work addresses Vision-Language Navigation in Continuous Environments (VLN-CE) by introducing Cog-GA, an LLM-based generative agent that simulates human-like cognition for navigation. Cog-GA builds a graph-based cognitive map as external memory, uses a waypoint predictor to constrain the search space, and employs a dual-channel (what/where) scene description to sharpen attention for sub-instruction execution. An instruction rationalization module and a reflection mechanism enable continual learning and adaptive replanning, while the memory stream integrates temporal, spatial, and semantic cues to inform planning. Experimental results on VLN-CE benchmarks show competitive SR and OSR, with ablations confirming the effectiveness of the cognitive map and instruction processing components, and the work advances interpretable, strategic VLN-CE agents, albeit with speed limitations in LLM communication.

Abstract

Vision Language Navigation in Continuous Environments (VLN-CE) represents a frontier in embodied AI, demanding agents to navigate freely in unbounded 3D spaces solely guided by natural language instructions. This task introduces distinct challenges in multimodal comprehension, spatial reasoning, and decision-making. To address these challenges, we introduce Cog-GA, a generative agent founded on large language models (LLMs) tailored for VLN-CE tasks. Cog-GA employs a dual-pronged strategy to emulate human-like cognitive processes. Firstly, it constructs a cognitive map, integrating temporal, spatial, and semantic elements, thereby facilitating the development of spatial memory within LLMs. Secondly, Cog-GA employs a predictive mechanism for waypoints, strategically optimizing the exploration trajectory to maximize navigational efficiency. Each waypoint is accompanied by a dual-channel scene description, categorizing environmental cues into 'what' and 'where' streams as the brain. This segregation enhances the agent's attentional focus, enabling it to discern pertinent spatial information for navigation. A reflective mechanism complements these strategies by capturing feedback from prior navigation experiences, facilitating continual learning and adaptive replanning. Extensive evaluations conducted on VLN-CE benchmarks validate Cog-GA's state-of-the-art performance and ability to simulate human-like navigation behaviors. This research significantly contributes to the development of strategic and interpretable VLN-CE agents.

Figures (26)

• Figure 1: In continuous environments, the Cog-GA agent conducts navigation based on the history chain retrieved from the cognitive map constructed during the navigation process, integrating observations to infer the next target index.
• Figure 2: This figure provides an overview of the Cog-GA agent. The panorama of the current position and the main instructions for the task are inputted into both the waypoint predictor module and the instruction processing module. This input is then processed to generate prompts for the LLMs-based high-level target planner, determining the target waypoint. The direction and angle of the target waypoint are subsequently input into the actuator to execute the actions.
• Figure 3: Overview of the agent's cognitive map retrieval methods. Blue nodes are spatially related, and orange nodes are object-related. Connections between blue nodes contain direction and angle data. B represents the history chain, and C the observation chain.
• Figure 4: Summary of the agent's inference process. The cognitive map in the memory stream is updated simultaneously.
• Figure 5: The agent's reflection process is based on ground-truth data. If a new reflection memory is considered non-redundant, it is stored in the memory stream along with its score.