Toward Scene Graph and Layout Guided Complex 3D Scene Generation

TL;DR

GraLa3D tackles the challenge of generating complex 3D scenes from text by marrying scene graphs with explicit layout guidance. It introduces a three-stage pipeline—Scene Graph Composition, Node-to-3D Generation (with single-object and super-node branches), and 3D Scene Harmonization—driven by an LLM and diffusion-based priors, reinforced by localization and masked ISM losses to prevent object entanglement. Quantitative and qualitative results show GraLa3D achieving higher CLIP alignment and more accurate interactions than state-of-the-art baselines, demonstrating improved scalability to scenes with many objects. The work enables coherent, interactive 3D scene synthesis from natural language with practical implications for gaming, VR, and autonomous simulation, while acknowledging limitations in mesh quality from the 3DGS representation and suggesting future improvements in 3D mesh reconstruction.

Abstract

Recent advancements in object-centric text-to-3D generation have shown impressive results. However, generating complex 3D scenes remains an open challenge due to the intricate relations between objects. Moreover, existing methods are largely based on score distillation sampling (SDS), which constrains the ability to manipulate multiobjects with specific interactions. Addressing these critical yet underexplored issues, we present a novel framework of Scene Graph and Layout Guided 3D Scene Generation (GraLa3D). Given a text prompt describing a complex 3D scene, GraLa3D utilizes LLM to model the scene using a scene graph representation with layout bounding box information. GraLa3D uniquely constructs the scene graph with single-object nodes and composite super-nodes. In addition to constraining 3D generation within the desirable layout, a major contribution lies in the modeling of interactions between objects in a super-node, while alleviating appearance leakage across objects within such nodes. Our experiments confirm that GraLa3D overcomes the above limitations and generates complex 3D scenes closely aligned with text prompts.

Figures (11)

• Figure 1: Method Overview of GraLa3D. The proposed method consists of three stages: (a) Scene Graph Composition, (b) Node-to-3D Generation, and (c) 3D Scene Harmonization. Stage (a) converts a text prompt $y^g$ into a scene graph $G$ representation with the associated layout bounding boxes $B$. With nodes corresponding to objects with or without interaction, Stage (b) generates 3DGS aligned with the information described in (a). Finally, Stage (c) enforces the output scene to exhibit proper appearance and texture consistency.
• Figure 2: Scene graph composition. We utilize LLM to construct a scene graph describing objects and their relations. In particular, nodes in blue denotes single objects in the scene, while supernodes in orange describe objects with interactions.
• Figure 3: Super-node Generation. Given $y^S_{1,2}$ as input, 3DGS models $\theta^S_1$, $\theta^S_2$ are initialized and optimized. In the upper branch, $\theta^S_1$, $\theta^S_2$ are jointly optimized using $y^S_{1,2}$ and $b^S_1 \cup b^S_2$. In the lower branch, taking the horse ($y^S_2$) as an example, the attention map $D^S_2$, which corresponds to the token of $y^S_2$ from $x^S_{1,2}$, is used as guidance to localize the generation region of $x^S_2$.
• Figure 4: Example text-to-3D generation with four objects. Given the text prompt of "a Wizard in front of a Wooden Desk, gazing into a Crystal Ball perched atop the Wooden Desk, with a Stack of Ancient Spell Books perched atop the Wooden Desk", we are able to generate a wizard-crustal ball pair with proper spatial and interaction relationship, with the ball and books placed on the table. On the other hand, GALA3D fails to generate the interaction between the wizard and the crystal ball, while GraphDreamer fails to produce a proper 3D layout (e.g., the books are larger than the table).
• Figure 5: Examples of text-to-3D generation with five objects. Given the prompt of "A mermaid sits on a coral throne, guarding a treasure chest on a stone while a sea turtle swims above her." and "A bear playing a saxophone stands on the stage, with a bar counter adjacent to it and a wooden wine cabinet filled with wine bottles behind the bar counter.", our GraLA3D is able to generate proper interactions of the mermaid-throne pair where the mermaid is sitting on the coral throne. Similarly, the bear-sax pair is also generated properly with the bear holding and playing the saxophone. Conversely, the mermaid generated by GALA3D is floating above the throne instead of sitting on it, and the bear's hand is not touching the saxophone at all. As for GraphDreamer, it fails to generate all five objects in both cases. We show their results by using only "mermaid sits on a coral throne" and "a bear playing saxophone" as input prompts.