Augmented Commonsense Knowledge for Remote Object Grounding

TL;DR

This work tackles remote object grounding in vision-language navigation by integrating augmented commonsense knowledge into a spatio-temporal knowledge graph. It retrieves and refines ConceptNet-based facts for detected objects, then fuses them with object features via a knowledge graph–aware cross-modal encoder and a concept history module to improve visual-text alignment and action reasoning. The approach, including a dedicated commonsense-based decision-making pipeline, achieves state-of-the-art results on REVERIE unseen and is supported by comprehensive ablations showing the importance of knowledge graph structure, temporal history, and controlled use of external knowledge. The findings demonstrate that incorporating structured commonsense and temporal context can significantly enhance navigation and grounding in unseen environments, suggesting broader applicability to knowledge-driven VLN tasks.

Abstract

The vision-and-language navigation (VLN) task necessitates an agent to perceive the surroundings, follow natural language instructions, and act in photo-realistic unseen environments. Most of the existing methods employ the entire image or object features to represent navigable viewpoints. However, these representations are insufficient for proper action prediction, especially for the REVERIE task, which uses concise high-level instructions, such as ''Bring me the blue cushion in the master bedroom''. To address enhancing representation, we propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as a spatio-temporal knowledge graph for improving agent navigation. Specifically, the proposed approach involves constructing a knowledge base by retrieving commonsense information from ConceptNet, followed by a refinement module to remove noisy and irrelevant knowledge. We further present ACK which consists of knowledge graph-aware cross-modal and concept aggregation modules to enhance visual representation and visual-textual data alignment by integrating visible objects, commonsense knowledge, and concept history, which includes object and knowledge temporal information. Moreover, we add a new pipeline for the commonsense-based decision-making process which leads to more accurate local action prediction. Experimental results demonstrate our proposed model noticeably outperforms the baseline and archives the state-of-the-art on the REVERIE benchmark.

Figures (5)

• Figure 1: Action prediction by the baseline and our method. Utilizing visible objects alongside commonsense knowledge as a spatio-temporal knowledge graph improves visual representation and action prediction. Best viewed in color.
• Figure 2: Main architecture of our proposed method. (a) Retrieving and refining the commonsense knowledge. (b) Initializing the concept history which represents the entire instruction and obtaining the text embedding. (c) ACK receives the detected objects, commonsense knowledge, and their temporal information to output weighted raw concept features which are utilized in the commonsense-based decision-making pipeline and the baseline model. (d) Inspired by the baseline agent, we add a new pipeline to produce the local action score and predict the object. Best viewed in color.
• Figure 3: Encoding the relative position of objects with respect to the heading and elevation angles of the agent.
• Figure 4: Visualization example of navigation performance for comparing ACK and the baseline method. We can see that our method predicts the correct action while DUET selects the wrong candidate direction. The concepts, including detected objects and retrieved commonsense knowledge, with the highest weights are used as landmarks in the instruction. Therefore, taking advantage of these concepts leads to visual representation enhancement and more accurate alignment between visual and textual information. Best viewed in color
• Figure 5: Learned concept-to-concept correlation matrix