ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models

TL;DR

ZING-3D tackles open-world, incremental 3D scene graph generation by leveraging pretrained Vision-Language Models to produce zero-shot open-vocabulary 2D graphs, then grounding them in 3D via depth-based projection and segmentation. The system incrementally fuses observations into a global 3D graph, enriched with object descriptors, 3D coordinates, room context, and relations with inter-object distances. It also introduces task-guided pruning to focus reasoning for Vision-Language Navigation and object-goal tasks. Experimental results on Replica and HM3D show high accuracy in node and edge labeling without task-specific training, demonstrating practical potential for robotics applications and open-world scene understanding.

Abstract

Understanding and reasoning about complex 3D environments requires structured scene representations that capture not only objects but also their semantic and spatial relationships. While recent works on 3D scene graph generation have leveraged pretrained VLMs without task-specific fine-tuning, they are largely confined to single-view settings, fail to support incremental updates as new observations arrive and lack explicit geometric grounding in 3D space, all of which are essential for embodied scenarios. In this paper, we propose, ZING-3D, a framework that leverages the vast knowledge of pretrained foundation models to enable open-vocabulary recognition and generate a rich semantic representation of the scene in a zero-shot manner while also enabling incremental updates and geometric grounding in 3D space, making it suitable for downstream robotics applications. Our approach leverages VLM reasoning to generate a rich 2D scene graph, which is grounded in 3D using depth information. Nodes represent open-vocabulary objects with features, 3D locations, and semantic context, while edges capture spatial and semantic relations with inter-object distances. Our experiments on scenes from the Replica and HM3D dataset show that ZING-3D is effective at capturing spatial and relational knowledge without the need of task-specific training.

Figures (4)

• Figure 1: ZING-3D generates zero-shot open-vocabulary 3D scene graphs and incrementally updates them as a robot navigates an environment. (A) A Large Vision-Language Model outputs a 2D scene graph from a set of posed RGB images. (B) The 2D graph is projected to 3D, incorporating object coordinates, inter-object distances, and spatial relationships. (C) Each node contains object descriptions, associated room type, and object category.
• Figure 2: Architecture of ZING-3D: (A) An open-vocabulary 2D scene graph is built from a sequence of posed RGB images using a large vision-language model (VLM), capturing object descriptors, inter-object relations, room context, and object types. (B) The object list is filtered and only relevant relevant furniture and objects are extracted which are then passed to Grounded-SAM2 to generate bounding boxes and corresponding segmentation masks, which are then projected to 3D using depth maps. (C) The resulting 3D scene graph provides a structured, comprehensive representation of the scene, suitable for various downstream robotics tasks.
• Figure 3: An example of a generated 2D scene graph features.
• Figure 4: Example pruned scene graph for VLN Query - "Go near the sofa".