TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understanding

TL;DR

TESGNN tackles robust multi-view 3D scene understanding by enforcing symmetry through an ES GNN backbone and by temporally fusing local graphs with a graph-matching module to form a global scene representation. The ESGNN leverages FAN-GCL and EGCL layers to preserve equivariance under rotations and translations, improving accuracy and convergence, while the Temporal Graph Matching Network aligns embeddings across sequences via triplet representations and a contrastive objective. Experiments on 3DSSG/3RScan show superior recall for objects and relationships, faster training convergence, and competitive runtime, highlighting its potential for real-time robotic and vision applications. Overall, TESGNN offers a principled, scalable approach to robust multi-view 3D scene understanding with practical impact for autonomous systems.

Abstract

Scene graphs have proven to be highly effective for various scene understanding tasks due to their compact and explicit representation of relational information. However, current methods often overlook the critical importance of preserving symmetry when generating scene graphs from 3D point clouds, which can lead to reduced accuracy and robustness, particularly when dealing with noisy, multi-view data. Furthermore, a major limitation of prior approaches is the lack of temporal modeling to capture time-dependent relationships among dynamically evolving entities in a scene. To address these challenges, we propose Temporal Equivariant Scene Graph Neural Network (TESGNN), consisting of two key components: (1) an Equivariant Scene Graph Neural Network (ESGNN), which extracts information from 3D point clouds to generate scene graph while preserving crucial symmetry properties, and (2) a Temporal Graph Matching Network, which fuses scene graphs generated by ESGNN across multiple time sequences into a unified global representation using an approximate graph-matching algorithm. Our combined architecture TESGNN shown to be effective compared to existing methods in scene graph generation, achieving higher accuracy and faster training convergence. Moreover, we show that leveraging the symmetry-preserving property produces a more stable and accurate global scene representation compared to existing approaches. Finally, it is computationally efficient and easily implementable using existing frameworks, making it well-suited for real-time applications in robotics and computer vision. This approach paves the way for more robust and scalable solutions to complex multi-view scene understanding challenges. Our source code is publicly available at: https://github.com/HySonLab/TESGraph

Figures (7)

• Figure 1: A visualization of a multi-view scene graph from multiple 3D point cloud sequences. Our proposed TESGNN first generates local scene graphs for each sequence using Equivariant GNN. Then, the local scene graphs are merged by passing through a temporal layer to form a global scene graph representing the entire scene.
• Figure 2: Overview of our proposed TESGNN. Our approach takes sequences of point clouds a) as input to generate a geometric segmentation b). Subsequently, the properties of each segment and a neighbor graph between segments are constructed. The properties d) and neighbor graph e) of the segments that have been updated in the current frame c) are used as the inputs to compute node and edge features f) and to predict a 3D scene graph g). Then it goes through the temporal layer to fuse graphs from different sequences to a global one h).
• Figure 3: ESGNN Scene Graph Extractor pipeline. The model comprises two main layers: (1) Feature-wise Attention Graph Convolution Layer (FAN-GCL) and (2) Equivariant Graph Convolution Layer (EGCL). FAN-GCL handles large inputs with multi-head attention to update node and edge features, while EGCL ensures symmetry preservation by incorporating bounding box coordinates into the message-passing mechanism. ESGNN leverages these layers to maintain rotation and permutation equivariance, thus enhance the quality of scene graph generation.
• Figure 4: Temporal Graph Matching pipeline. First, node and edge embeddings derived from scene graphs of different sequences (a). For each sequence, edge embeddings are concatenated with the target node embeddings to create Predicate-Object embeddings (b), which then pass through a linear layer followed by sum pooling (c). For each node, the embeddings from all associated edges are concatenated and processed through another linear layer and self-attention mechanism to generate the final representation of each segment (d). These final Triplet Embeddings are utilized for top-K retrieval graph matching (e).
• Figure 5: Comparison of recall result over training steps, column-wise interpretation. (a), (d) illustrates the result of our model versus SGFN. (b), (e) shows the ablation study. (c), (f) shows the result while applying our model with the image encoder.