NavHint: Vision and Language Navigation Agent with a Hint Generator

TL;DR

NavHint addresses the gap in vision-language navigation where navigation losses alone fail to foster deep visual semantics. It introduces a Transformer-based hint generator that, at each navigation step, outputs descriptive hints consisting of sub-instruction progress, landmark ambiguity, and targeted distinctive objects, trained via a synthetic hint dataset. The approach achieves strong performance on R2R and R4R benchmarks while enhancing interpretability of agent actions. This hint-based indirect supervision provides a practical pathway to richer cross-modal grounding in VLN systems.

Abstract

Existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment. In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions. The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent's attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment. We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the interpretability of the agent's actions.

Figures (8)

• Figure 1: Given the instruction and three candidate viewpoints, the navigation agent with the assistance of the hint generator, produces descriptions of the visual environment with three key elements: sub-instruction, landmark ambiguity and targeted distinctive objects.
• Figure 2: Navigation Hint Dataset. An example of a navigation hints with the landmark ambiguity of "Missing Landmarks". The sub-instruction is"walk into the hallway"(), and the landmark "hallway" () in the instruction is observed in the view1 rather than target view3, which can potentially mislead the navigation agent. The target distinctive objects "wooden dining table" and "marble countertop."() are then provided. "Blue walls" () is non-distinctive as it appears in both view2 and view3.
• Figure 3: Statistics of different categories of landmark ambiguity.
• Figure 4: Model Architecture. We introduce a hint generator designed to help the navigation agent acquire a deep understanding of the visual environment. The weighted vision representations (), used as image prefix, and the instruction text representation, used as instruction prefix (), are input into a GPT2 decoder. The decoder generates hints during navigation at each step. The hints include the three parts of sub-instruction (), landmark ambiguity (), and target distinctive objects ().
• Figure 5: Accuracy of the generated landmark ambiguity. Sub.: Sub-instruction.