VL-KnG: Visual Scene Understanding for Navigation Goal Identification using Spatiotemporal Knowledge Graphs

TL;DR

VL-KnG tackles persistent scene understanding for navigation by building a spatiotemporal knowledge graph from video chunks and using GraphRAG-enabled query processing for goal localization. It introduces semantic-based object association across chunks to maintain identity and a comprehensive WalkieKnowledge benchmark to evaluate spatial reasoning, object search, and relational queries. Real-world hardware experiments show performance competitive with state-of-the-art VLMs while providing explainable graph-based reasoning and computational efficiency suitable for real-time deployment. The approach offers a practical framework for robust, interpretable navigation across localization, planning, and execution tasks, with future work extending dynamic environments and multimodal reasoning.

Abstract

Vision-language models (VLMs) have shown potential for robot navigation but encounter fundamental limitations: they lack persistent scene memory, offer limited spatial reasoning, and do not scale effectively with video duration for real-time application. We present VL-KnG, a Visual Scene Understanding system that tackles these challenges using spatiotemporal knowledge graph construction and computationally efficient query processing for navigation goal identification. Our approach processes video sequences in chunks utilizing modern VLMs, creates persistent knowledge graphs that maintain object identity over time, and enables explainable spatial reasoning through queryable graph structures. We also introduce WalkieKnowledge, a new benchmark with about 200 manually annotated questions across 8 diverse trajectories spanning approximately 100 minutes of video data, enabling fair comparison between structured approaches and general-purpose VLMs. Real-world deployment on a differential drive robot demonstrates practical applicability, with our method achieving 77.27% success rate and 76.92% answer accuracy, matching Gemini 2.5 Pro performance while providing explainable reasoning supported by the knowledge graph, computational efficiency for real-time deployment across different tasks, such as localization, navigation and planning. Code and dataset will be released after acceptance.

Figures (5)

• Figure 1: Real-world deployment examples of VL-KnG for robot navigation. The system processes natural language queries to identify goal objects and provides pose estimates for navigation planning. In each case, the robot's perspective on detected objects and spatial relationships shows the system's ability to maintain scene understanding across temporal sequences.
• Figure 2: VL-KnG system architecture showing the complete pipeline from video frame input to navigation goal localization. In Phase 1, the environment knowledge graph is built iteratively using a source tour video. In Phase 2, the actual query processing and goal frame identification are performed. Assuming that the tour video is paired with robot poses, the corresponding pose is sent as a goal for the navigation system in Phase 3.
• Figure 3: VL-KnG employs a two-stage prompt template pipeline for spatiotemporal knowledge graph construction from video data. The first stage (Chunk Graph Construction) processes video chunks using modern vision-language models to detect objects and spatial relationships while assigning unique identifiers across frames. The second stage (Spatiotemporal Object Association) employs semantic-based association mechanisms that leverage large language model reasoning to align chunk-level graphs with a global knowledge representation, maintaining object identity.
• Figure 4: The WalkieKnowledge Benchmark includes $\sim200$ questions across 8 trajectories, with question types distributed according to the environment (indoor/outdoor).
• Figure 5: Examples from the Walkie-Knowledge Dataset, covering diverse indoor and outdoor environments such as shopping malls, supermarkets, exhibitions, bazaars, and streets.