Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval

Haozhou Li, Xiangyu Dong, Huiyan Jiang, Yaoming Zhou, Xiaoguang Ma

TL;DR

This work proposes CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases.

Abstract

Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.

CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval

TL;DR

This work proposes CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases.

Abstract

Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.
Paper Structure (15 sections, 5 equations, 4 figures, 3 tables)

This paper contains 15 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Vision-and-Language Navigation (VLN)
  4. Memory-Augmented and Reflective Agents
  5. METHODOLOGY
  6. Multimodal Experience Memory (MEM)
  7. Retrieval-Augmented Generation Pipeline (RAGP)
  8. Reflection and Memory Update
  9. EXPERIMENTS AND ANALYSIS
  10. Experiment Setup
  11. Simulation Experiments
  12. Ablation Study
  13. Case Study
  14. Experiments on A Real Robot
  15. CONCLUSIONS

Figures (4)

  • Figure 1: The overall CMMR-VLN framework consists of three modules from left to right. The Multimodal Experience Memory (MEM) performs memory building before navigation. The Retrieval-Augmented Generation Pipeline (RAGP) carries out corresponding prompting and action execution at each navigation step. The Reflection Module conducts reflection and updates the experience in memory after navigation.
  • Figure 2: Details of the Reflection Module in Fig 1.
  • Figure 3: Case study 1.
  • Figure 4: Case study 2.