MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains

TL;DR

This work tackles Vision-and-Language Navigation by removing heavy training requirements and enhancing interpretability through a Memory Topological Map and a Navigation Chain of Thoughts. The MC-GPT framework uses a memory-augmented topological memory, a demonstration-based reasoning module, and a prompt-managed pipeline to let an LLM steer navigation, supplemented by perception and action modules. Empirical results on REVERIE and R2R show clear gains over training-based baselines and competitive performance against other map-based LLM approaches, with notable improvements in interpretability of the reasoning process. The approach promises practical benefits in generalization, efficiency, and transparency for multimodal navigation tasks.

Abstract

In the Vision-and-Language Navigation (VLN) task, the agent is required to navigate to a destination following a natural language instruction. While learning-based approaches have been a major solution to the task, they suffer from high training costs and lack of interpretability. Recently, Large Language Models (LLMs) have emerged as a promising tool for VLN due to their strong generalization capabilities. However, existing LLM-based methods face limitations in memory construction and diversity of navigation strategies. To address these challenges, we propose a suite of techniques. Firstly, we introduce a method to maintain a topological map that stores navigation history, retaining information about viewpoints, objects, and their spatial relationships. This map also serves as a global action space. Additionally, we present a Navigation Chain of Thoughts module, leveraging human navigation examples to enrich navigation strategy diversity. Finally, we establish a pipeline that integrates navigational memory and strategies with perception and action prediction modules. Experimental results on the REVERIE and R2R datasets show that our method effectively enhances the navigation ability of the LLM and improves the interpretability of navigation reasoning.

Figures (7)

• Figure 1: Accurate memory and suitable strategies are crucial for navigation tasks. A topological map can precisely record information about viewpoints, while incorporating human navigation reasoning examples can enhance the diversity of strategies employed by the LLM.
• Figure 2: Overview of our pipeline. The Memory Topological Map module manages a map for storing memory, while the Navigation Chain of Thoughts module provides a demonstration example with a reasoning chain. The Prompt manager organizes all essential information and feeds it to the LLM. The LLM then makes a viewpoint selection and employs a Shortest-path Planner to navigate to the selected viewpoint.
• Figure 3: The construction of the Memory Topological Map. Initially, with object detection and image captioning, we can convert visual perceptions into textual descriptions. Subsequently, we construct a map comprising viewpoints and objects. Notably, the red node represents the viewpoint where the agent is currently located, while the yellow nodes denote unvisited viewpoints, and the blue nodes mean visited viewpoints. The squares on the map represent objects, and squares with the same colors indicate that they are clustered together. Finally, we utilize a textual prompt template to interpret the map.
• Figure 4: Two steps in the Navigation Chain of Thoughts: CoT mining and CoT querying. CoT mining aims to construct an example set comprising diverse reasoning chains, while CoT querying aims to select a similar example with an appropriate reasoning chain for the LLM's navigation.
• Figure 5: Our prompt manager integrates two types of prompts, the $Prefix prompt and the $Input prompt. The former is designed to illustrate basic reasoning rules, while the latter provides perceptual information. The prompt manager is employed for both CoT mining and navigation inference. Thus, the symbol $\ast$ denotes content exclusively relevant during CoT mining, whereas $\ast\ast$ means content exclusive to navigation inference.