SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequences

TL;DR

SceneLinker addresses the challenge of generating coherent, compositionally accurate 3D scenes from RGB sequences by predicting a global 3D scene graph and synthesizing a 3D scene via a Graph-VAE with a Joint Shape and Layout (JSL) block. It introduces a Cross-Check Feature Attention (CCFA) GCN to robustly propagate node-edge information, and extends scene graphs with CLIP-informed context and DeepSDF shape priors for reliable layout-shape fusion. The approach yields state-of-the-art performance on 3RScan/3DSSG and SG-FRONT in both scene-graph prediction and 3D generation, while offering faster inference than diffusion-based methods. Practically, SceneLinker enables efficient, space-aware MR content creation and collaborative spatial authoring from real-world environments, with potential extensions to texture mapping and user-centric MR studies.

Abstract

We introduce SceneLinker, a novel framework that generates compositional 3D scenes via semantic scene graph from RGB sequences. To adaptively experience Mixed Reality (MR) content based on each user's space, it is essential to generate a 3D scene that reflects the real-world layout by compactly capturing the semantic cues of the surroundings. Prior works struggled to fully capture the contextual relationship between objects or mainly focused on synthesizing diverse shapes, making it challenging to generate 3D scenes aligned with object arrangements. We address these challenges by designing a graph network with cross-check feature attention for scene graph prediction and constructing a graph-variational autoencoder (graph-VAE), which consists of a joint shape and layout block for 3D scene generation. Experiments on the 3RScan/3DSSG and SG-FRONT datasets demonstrate that our approach outperforms state-of-the-art methods in both quantitative and qualitative evaluations, even in complex indoor environments and under challenging scene graph constraints. Our work enables users to generate consistent 3D spaces from their physical environments via scene graphs, allowing them to create spatial MR content. Project page is https://scenelinker2026.github.io.

Figures (6)

• Figure 1: Given (A) RGB sequences as input to SceneLinker, the system estimates incremental (B.1) 3D entities. It then computes the node and edge features for each 3D segment to extract graph properties and propagates them to predict the (B.2) global scene graph. The (C.1) predicted scene graph and SDF shape code are embedded into a (C.2) GCN-based VAE, which fuses the shape and layout to finally generate a 3D scene.
• Figure 2: Configuration of (a) extended scene graph and proposed (b) graph-VAE sturcture for encoding, decoding.
• Figure 3: Qualitative comparisons with other generative models. Our method is compared with approaches that utilize text-based graph. For the challenging scene graph constraints (close by, symmetrical), correctly generated areas are highlighted in green, while incorrectly generated areas are highlighted in red.
• Figure 4: Visual comparison of the real and virtual scene. To capture the entire compositional arrangement of the real-world environment, we visualize the real scene above using point clouds included in 3RScan.
• Figure 5: SceneLinker is designed to operate in actual MR environments, showcasing its potential for deployment in AR/VR applications such as remote spatial sharing and collaborative editing.