Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents

Xunyi Zhao, Gengze Zhou, Qi Wu

TL;DR

This work tackles evaluating Multimodal Large Language Models (MLLMs) as embodied agents in Vision-and-Language Navigation (VLN). It introduces VLN-MME, a simulator-free, modular framework that decouples Model, Task, and Agent to enable scalable, diagnostic evaluations and component-level ablations. Across a stratified, representative benchmark drawn from R2R, REVERIE, and ObjectNav, zero-shot MLLMs consistently lag behind specialized VLN baselines, with proprietary models leading among closed systems; surprisingly, Chain-of-Thought and self-reflection often degrade performance due to limited historical grounding. The authors provide granular error analyses, revealing that failures are dominated by spatial understanding and perception-action grounding rather than language generation, and demonstrate the utility of oracle guidance and failure-aware prompts for diagnostic study. Overall, VLN-MME offers a practical path for post-training data generation and evaluation protocol design to advance embodied capabilities in MLLMs while furnishing reproducible artifacts for the research community.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a wide range of vision-language tasks. However, their performance as embodied agents, which requires multi-round dialogue spatial reasoning and sequential action prediction, needs further exploration. Our work investigates this potential in the context of Vision-and-Language Navigation (VLN) by introducing a unified and extensible evaluation framework to probe MLLMs as zero-shot agents by bridging traditional navigation datasets into a standardized benchmark, named VLN-MME. We simplify the evaluation with a highly modular and accessible design. This flexibility streamlines experiments, enabling structured comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. Crucially, enabled by our framework, we observe that enhancing our baseline agent with Chain-of-Thought (CoT) reasoning and self-reflection leads to an unexpected performance decrease. This suggests MLLMs exhibit poor context awareness in embodied navigation tasks; although they can follow instructions and structure their output, their 3D spatial reasoning fidelity is low. VLN-MME lays the groundwork for systematic evaluation of general-purpose MLLMs in embodied navigation settings and reveals limitations in their sequential decision-making capabilities. We believe these findings offer crucial guidance for MLLM post-training as embodied agents.

VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents

TL;DR

This work tackles evaluating Multimodal Large Language Models (MLLMs) as embodied agents in Vision-and-Language Navigation (VLN). It introduces VLN-MME, a simulator-free, modular framework that decouples Model, Task, and Agent to enable scalable, diagnostic evaluations and component-level ablations. Across a stratified, representative benchmark drawn from R2R, REVERIE, and ObjectNav, zero-shot MLLMs consistently lag behind specialized VLN baselines, with proprietary models leading among closed systems; surprisingly, Chain-of-Thought and self-reflection often degrade performance due to limited historical grounding. The authors provide granular error analyses, revealing that failures are dominated by spatial understanding and perception-action grounding rather than language generation, and demonstrate the utility of oracle guidance and failure-aware prompts for diagnostic study. Overall, VLN-MME offers a practical path for post-training data generation and evaluation protocol design to advance embodied capabilities in MLLMs while furnishing reproducible artifacts for the research community.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a wide range of vision-language tasks. However, their performance as embodied agents, which requires multi-round dialogue spatial reasoning and sequential action prediction, needs further exploration. Our work investigates this potential in the context of Vision-and-Language Navigation (VLN) by introducing a unified and extensible evaluation framework to probe MLLMs as zero-shot agents by bridging traditional navigation datasets into a standardized benchmark, named VLN-MME. We simplify the evaluation with a highly modular and accessible design. This flexibility streamlines experiments, enabling structured comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. Crucially, enabled by our framework, we observe that enhancing our baseline agent with Chain-of-Thought (CoT) reasoning and self-reflection leads to an unexpected performance decrease. This suggests MLLMs exhibit poor context awareness in embodied navigation tasks; although they can follow instructions and structure their output, their 3D spatial reasoning fidelity is low. VLN-MME lays the groundwork for systematic evaluation of general-purpose MLLMs in embodied navigation settings and reveals limitations in their sequential decision-making capabilities. We believe these findings offer crucial guidance for MLLM post-training as embodied agents.
Paper Structure (58 sections, 19 figures, 6 tables)

This paper contains 58 sections, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Evaluating MLLMs in Spatial and Embodied Contexts
  4. MLLMs for Vision-and-Language Navigation
  5. Method
  6. A Modular Framework for VLN Evaluation
  7. Model
  8. Task
  9. Agent
  10. Extensibility and Orchestration
  11. Dataset Construction for Efficient Evaluation
  12. Simulator-Free Environment Design
  13. Experiments
  14. Settings
  15. Evaluation Metrics.
  16. ...and 43 more sections

Figures (19)

  • Figure 1: A high-level structure for the benchmark, centered on the interplay between Tasks, Agents, and Models.
  • Figure 2: The composition of the VLN-MME.
  • Figure 3: Performance comparison of agents using text summarization memory under different reasoning strategies across multiple backbone MLLMs.
  • Figure 4: A high-level analysis of success and failure modes for Qwen2.5-VL-7B model using an agent with text map memory.
  • Figure 5: GPT5 prompt for generating marker caption
  • ...and 14 more figures