Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation

TL;DR

This paper argues that VLN evaluation has been overly coarse and proposes a fine-grained framework based on a context-free grammar (CFG) to decompose instructions into atomic categories. The NavNuances dataset, built from 90 Matterport environments, covers five atomic instruction categories (Direction Change, Vertical Movement, Numerical Comprehension, Landmark Recognition, Region Recognition) and uses an iterative CFG-LMM workflow with human refinement. Benchmarking diverse models, including zero-shot GPT-4-vision variants, reveals strong gains in directional and landmark tasks but persistent challenges in numerical comprehension and region understanding, as well as biases in turning decisions. The work highlights the need for better integration of numeric, layout, and directional reasoning and suggests avenues for automated CFG induction to advance VLN evaluation and system design.

Abstract

This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems.

Figures (18)

• Figure 1: Examples of constructed interventions for VLN instructions. Example 1 demonstrates an intervention related to directional concepts, while Example 2 focuses on landmarks. Nonetheless, a subset of the model's predictions remains unchanged following the intervention, suggesting a deficiency in the model's ability to grasp underlying concepts.
• Figure 2: Schematic diagram of annotation criteria for five main categories in the NavNuances dataset.
• Figure 3: The success rate of models evaluated on five main categories of NavNuances dataset. Human performance is denoted by the green dashed line.
• Figure 4: Success rate relative to two additional random agents in the numerical comprehension category. Agent 1* is the random agent that knows the concept of entering the room in the corridor. Agent 2* is the random agent which also has directional awareness. The success rates of Agent 1* and Agent 2* are 32.06% and 41.03%.
• Figure 5: Results of two subsets of the landmark recognition category in the NavNuances dataset. The significant gap of the 'moving towards' subset comes from large pre-trained vision models since NavGPT3.5