Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation

TL;DR

Build-A-Scene tackles the challenge of interactive 3D layout control for diffusion-based T2I generation by reframing image synthesis as a multi-stage process over stages $i \in [0,n]$. It introduces Dynamic Self-Attention (DSA) to seamlessly insert new objects while preserving prior content and a consistent 3D translation strategy to maintain object identity when layouts change, all within a training-free, depth-conditioned framework leveraging 3D boxes. The approach demonstrates 2× improvements in object-generation success rate over depth-conditioned baselines and superior preservation under layout edits compared with 2D-layout methods, validated through both quantitative metrics (OA, mIoU, CLIP$_{\mathrm{T2I}}$) and qualitative assessments. These results suggest a practical path for interactive 3D scene design with diffusion models, enabling iterative refinement, proper depth perception, and coherent object interactions such as shadows and reflections.

Abstract

We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control. Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects' placement and relationships from text descriptions. Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes. This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control. We replace the traditional 2D boxes used in layout control with 3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages. We achieve this through our proposed Dynamic Self-Attention (DSA) module and the consistent 3D object translation strategy. Experiments show that our approach can generate complicated scenes based on 3D layouts, boosting the object generation success rate over the standard depth-conditioned T2I methods by 2x. Moreover, it outperforms other methods in comparison in preserving objects under layout changes. Project Page: \url{https://abdo-eldesokey.github.io/build-a-scene/}

Figures (8)

• Figure 1: Build-A-Scene is an interactive diffusion-based approach for image generation based on a user-provided 3D layout. At each generation stage, the user can control object type, location, and orientation (in-plane rotation) in 3D. Build-A-Scene ensures that objects are seamlessly integrated into the scene (see shadows and reflections) and preserve their identity under layout changes.
• Figure 2: Existing 2D layout control approaches, i.e., Layout Guidance ca_guidance accept static 2D layouts with no mechanism for 3D control or interactively changing the layout while preserving the image. LooseControl bhat2023loosecontrol can generate images conditioned on 3D boxes but cannot interactively move objects or handle a layout with diverse objects. Build-A-Scene is the first approach to support 3D layouts and allows users to manipulate them interactively.
• Figure 3: An overview of our pipeline Build-A-Scene (BAS). We formulate the Text-to-Image (T2I) task as a multi-stage generation process. Here, we illustrate a 3-stage scene. At stage$0$, the user defines an empty scene with full control over scene size and camera parameters. At stage$1$, the user adds an object (a white sofa) by adding a box and its corresponding prompt to the layout. Our proposed Dynamic Self-Attention (DSA) module, coupled with latent blending, ensures that the object is seamlessly integrated into the scene while preserving the existing contents. At stage$2$, we illustrate our Consistent 3D Translation strategy that allows moving the object in 3D while preserving its identity.
• Figure 4: A comparison between existing self-attention mechanisms and our proposed Dynamic Self-Attention (DSA).
• Figure 5: A qualitative comparison on the task 3D layout control.