Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D

Zirui Wang, Ruiping Liu, Yufan Chen, Junwei Zheng, Weijia Fan, Kunyu Peng, Di Wen, Jiale Wei, Jiaming Zhang, Rainer Stiefelhagen

TL;DR

A training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) with retrieval-augmented generation (RAG) for semantic scene graph generation, which enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework.

Abstract

3D scene graphs provide a structured representation of object entities and their relationships, enabling high-level interpretation and reasoning for robots while remaining intuitively understandable to humans. Existing approaches for 3D scene graph generation typically combine scene reconstruction with graph neural networks (GNNs). However, such pipelines require multi-modal data that may not always be available, and their reliance on heuristic graph construction can constrain the prediction of relationship triplets. In this work, we introduce a Scene Graph Retrieval-Reasoning Model in 3D (SGR3 Model), a training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) for semantic scene graph generation. SGR3 Model bypasses the need for explicit 3D reconstruction. Instead, it enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework. To improve retrieval robustness, we further introduce a weighted patch-level similarity selection mechanism that mitigates the negative impact of blurry or semantically uninformative regions. Experiments demonstrate that SGR3 Model achieves competitive performance compared to training-free baselines and on par with GNN-based expert models. Moreover, an ablation study on the retrieval module and knowledge base scale reveals that retrieved external information is explicitly integrated into the token generation process, rather than being implicitly internalized through abstraction.

SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D

TL;DR

A training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) with retrieval-augmented generation (RAG) for semantic scene graph generation, which enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework.

Abstract

3D scene graphs provide a structured representation of object entities and their relationships, enabling high-level interpretation and reasoning for robots while remaining intuitively understandable to humans. Existing approaches for 3D scene graph generation typically combine scene reconstruction with graph neural networks (GNNs). However, such pipelines require multi-modal data that may not always be available, and their reliance on heuristic graph construction can constrain the prediction of relationship triplets. In this work, we introduce a Scene Graph Retrieval-Reasoning Model in 3D (SGR3 Model), a training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) for semantic scene graph generation. SGR3 Model bypasses the need for explicit 3D reconstruction. Instead, it enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework. To improve retrieval robustness, we further introduce a weighted patch-level similarity selection mechanism that mitigates the negative impact of blurry or semantically uninformative regions. Experiments demonstrate that SGR3 Model achieves competitive performance compared to training-free baselines and on par with GNN-based expert models. Moreover, an ablation study on the retrieval module and knowledge base scale reveals that retrieved external information is explicitly integrated into the token generation process, rather than being implicitly internalized through abstraction.
Paper Structure (17 sections, 10 equations, 5 figures, 4 tables)

This paper contains 17 sections, 10 equations, 5 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. 3D Scene Graph Generation
  4. Retrieval Augmented Generation
  5. METHODOLOGY
  6. External Knowledge Base Building
  7. Key Frame Images Filtering
  8. Retrieval for Reference Edges
  9. Window-level 3D Scene Graph Generation
  10. Experiment
  11. Implementation Details
  12. Evaluation Metrics
  13. Comparison with Other Methods
  14. Ablation Study
  15. Inference on ScanNet
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Comparison of SGR3 Model and reconstruction-based models. The requirements of the reconstruction-based models include RGB images, depth information, camera poses, extrinsics, and intrinsics, whereas SGR3 Model requires only RGB images with information from an external knowledge base. Reconstruction-based pipelines often depend on geometric proximity heuristics to define candidate edges, thereby constraining relation modeling to spatially local interactions.
  • Figure 2: (Top) Traditional scene graph generation based on 3D reconstruction and GNN. (Bottom) Our training-free 3D scene graph generation pipeline. We split the complete RGB sequence into RGB windows containing several consecutive frames. Through ColQwen retrieval, we determine whether a single frame is a key frame whose visual content is not in the processed buffer. The RAG component performs patch embedding and knowledge base search.
  • Figure 3: Retrieval process for reference edge selection.
  • Figure 4: Visualization of a 3D scene graph generated by the SGR3 Model on ScanNet. Red dotted lines indicate incorrect predictions.
  • Figure 5: Attention Distribution when generating two predicates 'supported by' for triplets. Red vertical lines indicate the reference triplets span. Colors indicate relative attention strength. Several tokens within the span receive noticeable attention, top-2 corresponding tokens for each predicate are 'supported',':' and 'supported', 'supported'.