Object-Centric Representation Learning for Enhanced 3D Scene Graph Prediction

TL;DR

The paper tackles the bottleneck in 3D semantic scene graph prediction where object representation quality limits relationship reasoning. It proposes a two-stage, object-centric framework: a discriminative object feature encoder pretrained with cross-modal contrastive signals (images and text) and a Relationship Feature Encoder augmented with geometric cues, integrated into a Graph Neural Network with Bidirectional Edge Gating and Global/Local Spatial Enhancements. The approach achieves state-of-the-art results on the 3DSSG benchmark, with consistent object-classification gains driving improvements in predicate and triplet predictions, and demonstrates that plugging the pretrained encoder into existing models yields cross-model benefits. The work offers a practical, transfer-friendly blueprint for robust 3D scene graphs with strong implications for robotics and AR/VR applications, and outlines clear avenues for extending to incremental/object-detection-integrated or open-vocabulary settings.

Abstract

3D Semantic Scene Graph Prediction aims to detect objects and their semantic relationships in 3D scenes, and has emerged as a crucial technology for robotics and AR/VR applications. While previous research has addressed dataset limitations and explored various approaches including Open-Vocabulary settings, they frequently fail to optimize the representational capacity of object and relationship features, showing excessive reliance on Graph Neural Networks despite insufficient discriminative capability. In this work, we demonstrate through extensive analysis that the quality of object features plays a critical role in determining overall scene graph accuracy. To address this challenge, we design a highly discriminative object feature encoder and employ a contrastive pretraining strategy that decouples object representation learning from the scene graph prediction. This design not only enhances object classification accuracy but also yields direct improvements in relationship prediction. Notably, when plugging in our pretrained encoder into existing frameworks, we observe substantial performance improvements across all evaluation metrics. Additionally, whereas existing approaches have not fully exploited the integration of relationship information, we effectively combine both geometric and semantic features to achieve superior relationship prediction. Comprehensive experiments on the 3DSSG dataset demonstrate that our approach significantly outperforms previous state-of-the-art methods. Our code is publicly available at https://github.com/VisualScienceLab-KHU/OCRL-3DSSG-Codes.

Figures (8)

• Figure 1: (a) VL-SAT wang2023vl embeds object features non-discriminatively, leading to low-confidence predictions and frequent object misclassifications, which degrade relationship accuracy. In contrast, (b) our method embeds object features in a more discriminative manner, yielding high confidence scores and more accurate object classifications. Consequently, relationship predictions are significantly improved, resulting in a more coherent and semantically accurate scene graph.
• Figure 2: Histogram of object classification entropy and predicate prediction error rate, illustrating that higher entropy is associated with increased predicate errors under comparable relationship frequencies.
• Figure 3: Architecture of our Object Feature Encoder. The encoder extracts object embedding $\mathbf{z}^t$ from point clouds via affine transformation, aligned with CLIP features: $\mathbf{z}_{\text{text}}$ from text description and $\mathcal{Z}_I$, a set of image features from multiple 2D RGB images.
• Figure 4: Architecture overview. Object embeddings {$\mathbf{z}_i^t$, … , $\mathbf{z}_j^t$} are refined via Global Spatial Enhancement to incorporate global spatial context based on inter-object distances, producing enhanced features {$\bar{\mathbf{z}}_i^t$, … , $\bar{\mathbf{z}}_j^t$}. Simultaneously, the Local Spatial Enhancement locally preserves geometric relationships between object pairs. The Bidirectional Gated Graph Attention Network then selectively modulates the information of reverse edges, effectively capturing asymmetric relationships between objects.
• Figure 5: 3D scene graph visualizations.$\bm{\rightarrow}$ indicates true positive relations that are correctly predicted. $\bm{\rightarrow}$ denotes false positives, where the model predicts an incorrect predicate for an existing relation. $\bm{\dashrightarrow}$ represents either false negatives—missed ground-truth relations—or hallucinated relations that do not exist in the ground truth.

Theorems & Definitions (4)

• Proposition 1
• proof
• Proposition 2
• proof