Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph

TL;DR

This work addresses Visual Language Navigation (VLN) under coarse-grained instructions by introducing a VLN-specific event knowledge graph (VLN-EventKG) and a two-level planning framework (EventNav) that leverages large language models for subtask planning and a small model for action execution. It constructs VLN-EventKG by extracting event sequences from multiple VLN benchmarks, enabling retrieval-driven guidance to constrain LLM planning and produce coherent subtasks aligned with the VLN domain. A dynamic backtracking mechanism monitors subtask completion probability to trigger re-planning in real time, mitigating error accumulation during execution. Empirical results on R2R, REVERIE, and ALFRED show over 5% average improvements in success rate and demonstrate the value of combining event knowledge with hierarchical planning for robust coarse-grained VLN.

Abstract

Visual language navigation (VLN) is one of the important research in embodied AI. It aims to enable an agent to understand the surrounding environment and complete navigation tasks. VLN instructions could be categorized into coarse-grained and fine-grained commands. Fine-grained command describes a whole task with subtasks step-by-step. In contrast, coarse-grained command gives an abstract task description, which more suites human habits. Most existing work focuses on the former kind of instruction in VLN tasks, ignoring the latter abstract instructions belonging to daily life scenarios. To overcome the above challenge in abstract instruction, we attempt to consider coarse-grained instruction in VLN by event knowledge enhancement. Specifically, we first propose a prompt-based framework to extract an event knowledge graph (named VLN-EventKG) for VLN integrally over multiple mainstream benchmark datasets. Through small and large language model collaboration, we realize knowledge-enhanced navigation planning (named EventNav) for VLN tasks with coarse-grained instruction input. Additionally, we design a novel dynamic history backtracking module to correct potential error action planning in real time. Experimental results in various public benchmarks show our knowledge-enhanced method has superiority in coarse-grained-instruction VLN using our proposed VLN-EventKG with over $5\%$ improvement in success rate. Our project is available at https://sites.google.com/view/vln-eventkg

Figures (6)

• Figure 1: For Conventional VLN tasks, fine-grained task descriptions are provided to the agent. However, in the real world, humans often only provide coarse-grained task descriptions
• Figure 2: The construction pipeline of the event knowledge graph.
• Figure 3: The framework of our proposed EventNav with large-small-model collaboration using event knowledge.
• Figure 4: Subtask generation based on event knowledge (VLN-EventKG) retrieval and LLM using promoting learning.
• Figure 5: The training details of the dynamic backtracking mechanism.