TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation

TL;DR

This paper presents a VLN agent based on LLMs, exploring approaches to the zero-shot navigation problem and proposes the Thinking, Interacting, and Action (TINA) framework, which enables the agent to scrutinize perceptual information and autonomously query key clues within the environment through an introduced question-answering module.

Abstract

Zero-shot navigation is a critical challenge in Vision-Language Navigation (VLN) tasks, where the ability to adapt to unfamiliar instructions and to act in unknown environments is essential. Existing supervised learning-based models, trained using annotated data through reinforcement learning, exhibit limitations in generalization capabilities. Large Language Models (LLMs), with their extensive knowledge and emergent reasoning abilities, present a potential pathway for achieving zero-shot navigation. This paper presents a VLN agent based on LLMs, exploring approaches to the zero-shot navigation problem. To compensate for the shortcomings of LLMs in environmental perception, we propose the Thinking, Interacting, and Action (TINA) framework. TINA enables the agent to scrutinize perceptual information and autonomously query key clues within the environment through an introduced question-answering module, thereby aligning instructions with specific perceptual data. The navigation agent's perceptual abilities are enhanced through the TINA framework, while the explicit thought and query processes also improve the navigational procedure's explainability and transparency. We evaluate the performance of our method on the Room-to-Room dataset. The experiment results indicate that our approach improves the navigation performance of LLM-based agents. Our approach also outperformed some supervised learning-based methods, highlighting its efficacy in zero-shot navigation.

Figures (5)

• Figure 1: (a) A VLN Examples. (b) Supervised learning methods train a policy network based on pre-trained visual encoders and text embeddings. (c) LLM-based agents utilize an LLMs for reasoning. Our framework introduces additional modules to enhance the agent's capabilities. The snowflake symbol indicates frozen parameters, the flame signifies trainable ones.
• Figure 2: The schematic diagram of the TINA framework. It primarily consists of the core LLM-based agent and three peripheral modules. 1) The Visual Perception module is used to acquire descriptions of the surrounding environment and distance information related to various objects. 2) The Question-Answering Interaction module performs targeted clue queries on Candidates' visual images based on the agent's reasoning Thoughts. 3) The Trajectory Memorizer summarizes observations and actions from this step, storing new memories in the memory bank.
• Figure 3: The figure shows how to obtain an object's distance.
• Figure 4: The key prompts and their structure for Agent navigation. The handwritten text is content generated by LLM. The numbers indicate the sequence in which the modules operate.
• Figure 5: Some candidate viewpoints selected by the agent based on the Thought, along with the corresponding QAI output.