RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation

TL;DR

RAG-3DSG tackles noise and inefficiency in open-vocabulary 3D scene graph construction by introducing re-shot guided uncertainty estimation and object-level Retrieval-Augmented Generation. The method combines dynamic downsampling, best-view re-shot rendering, and context-aware retrieval to refine node descriptions before linking them with expressive edge captions, forming a complete 3D scene graph. Empirical results on Replica show higher node and edge precision than baselines and substantial gains in mapping speed, validating the approach's robustness and practicality for robotics. This work advances open-vocabulary scene understanding by integrating uncertainty-aware caption refinement with retrieval augmentation at the object level, enabling more reliable robotics manipulation and navigation.

Abstract

Open-vocabulary 3D Scene Graph (3DSG) generation can enhance various downstream tasks in robotics, such as manipulation and navigation, by leveraging structured semantic representations. A 3DSG is constructed from multiple images of a scene, where objects are represented as nodes and relationships as edges. However, existing works for open-vocabulary 3DSG generation suffer from both low object-level recognition accuracy and speed, mainly due to constrained viewpoints, occlusions, and redundant surface density. To address these challenges, we propose RAG-3DSG to mitigate aggregation noise through re-shot guided uncertainty estimation and support object-level Retrieval-Augmented Generation (RAG) via reliable low-uncertainty objects. Furthermore, we propose a dynamic downsample-mapping strategy to accelerate cross-image object aggregation with adaptive granularity. Experiments on Replica dataset demonstrate that RAG-3DSG significantly improves node captioning accuracy in 3DSG generation while reducing the mapping time by two-thirds compared to the vanilla version.

Figures (6)

• Figure 1: An overview of the RAG-3DSG framework. The upper part illustrates common challenges in multi-view 3D scene graph generation. Our pipeline addresses these issues through (a) Multi-view RGB-D frames are segmented and fused into a global object list with point clouds and semantic embeddings. (b) Re-shot images are used to select the best-view caption, which is compared with crop captions to estimate uncertainty; clustering is applied and the top-1 cluster is retained for fusion. (c) Low-uncertainty objects form a retrieval document, while high-uncertainty objects leverage retrieved context for caption refinement via a VLM. (d) Finally, spatial and semantic relationships among objects are inferred by an LLM to construct the 3D scene graph.
• Figure 2: Visualization of our 3DSGs for Replica straub2019replica Room 0 (left) and 1 (right). The blue points represent the objects and the red lines indicate the relationships between them.
• Figure 3: Ablation study quantifying the contribution of each pipeline component. Performance degradation is observed when removing re-shot guided uncertainty estimation (w/o Reshot), RAG (w/o RAG), concatenated re-shot images (w/o Concat), or using random retrieval (Random RAG), confirming the complementary roles of all proposed components.
• Figure 4: Examples of re-shot images automatically selected by our method. For each object category, we present the top-4 rendered views ranked by our re-shot scoring strategy. From top to bottom: armchair, vase, cushion, sofa, artwork, and TV stand.
• Figure 5: Heatmaps of semantic segmentation performance across scenes for different ablation settings. Metrics include mIoU, mRecall, mPrecision, mF1score, and frequency-weighted mIoU (fMiou). The full pipeline serves as the baseline, while “w/o concat”, “w/o rag”, “random rag”, and “w/o reshot” show the impact of removing or modifying individual components.