Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships

TL;DR

Open3DSG tackles open-vocabulary 3D scene graph prediction from point clouds by distilling 2D vision-language knowledge into a 3D graph neural network. It performs a two-stage inference: open-vocabulary object querying using CLIP-aligned embeddings and relationship grounding via a prompted LLM conditioned on detected objects, enabling open-set relationships beyond fixed label sets. The method demonstrates competitive results on the 3DSSG benchmark, with strong performance on long-tail and descriptive classes, and enables practical downstream tasks such as 3D triplet localization, material prediction, and affordance reasoning. This approach integrates 3D geometry with grounded language models to advance open-vocabulary 3D scene understanding and reasoning capabilities.

Abstract

Current approaches for 3D scene graph prediction rely on labeled datasets to train models for a fixed set of known object classes and relationship categories. We present Open3DSG, an alternative approach to learn 3D scene graph prediction in an open world without requiring labeled scene graph data. We co-embed the features from a 3D scene graph prediction backbone with the feature space of powerful open world 2D vision language foundation models. This enables us to predict 3D scene graphs from 3D point clouds in a zero-shot manner by querying object classes from an open vocabulary and predicting the inter-object relationships from a grounded LLM with scene graph features and queried object classes as context. Open3DSG is the first 3D point cloud method to predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, making it possible to express rare as well as specific objects and relationships in the predicted 3D scene graph. Our experiments show that Open3DSG is effective at predicting arbitrary object classes as well as their complex inter-object relationships describing spatial, supportive, semantic and comparative relationships.

Figures (11)

• Figure 1: Open3DSG. We present Open3DSG the first approach for learning to predict open-vocabulary 3D scene graphs from 3D point clouds. The advantage of our method is that it can be queried and prompted for any instance in the scene, such as the TV and Wall, to predict fine-grained semantic descriptions of objects and relationships. By considering all instance pairs in the scene, we can reconstruct a complete explicit open-vocabulary 3D scene graph.
• Figure 2: Open3DSG overview. Given a point cloud and RGB-D images with their poses, we distill the knowledge of two vision-language models into our GNN. The nodes are supervised by the embedding of OpenSeg ghiasi_2022_eccv and the edges are supervised by the embedding of the InstructBLIP dai2023instructblip vision encoder. At inference time, we first compute the cosine similarity between object queries encoded by CLIP radford_2021_PMLR and our distilled 3D node features to infer the object classes. Then we use the edge embedding as well as the inferred object classes to predict relationships for pairs of objects using a Qformer & LLM from InstructBLIP.
• Figure 3: Supervision feature extraction. For each instance in the 3D point cloud, we select the top k frames for object and predicate supervision. For objects, we encode the frames using OpenSeg ghiasi_2022_eccv and aggregate the computed features over the projected points. For predicates, we identify object pairs in the frame, crop the image at multiple scales and compute the image feature with the BLIP dai2023instructblip image encoder. The features are aggregated over all crops. Finally, both object and predicate features are fused across the multiple views.
• Figure 4: Qualitative open-vocabulary 3D scene graph predictions. We show the top-1 predictions on ScanNet Dai_2017_CVPR from Open3DSG. The nodes are queried using the 3DSSG Wald_2020_CVPR 160 class label set, while the edges are generated directly from the graph-conditioned LLM.
• Figure A: ScanNet vs. 3RScan. We choose ScanNet over 3RScan / 3DSSG as a distillation dataset since the FOV of each frame is generally higher and more objects are visible in one frame.