Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers

TL;DR

LayouSyn tackles open-vocabulary scene layout generation by substituting proprietary LLMs with lightweight open-source language models to derive object descriptions from prompts, and coupling this with an aspect-ratio aware diffusion Transformer that generates bounding-box layouts conditioned on those descriptions. The two-stage approach first extracts a Description Set from the prompt and then performs conditional layout generation conditioned on the description set and prompt, operating directly in the bounding-box coordinate space. The method achieves state-of-the-art results on NSR-1K spatial/numerical benchmarks and on COCO-GR layout quality, with GRIT pretraining further boosting performance. It also demonstrates practical applications in LLM-based initialization and automatic image editing via object addition, highlighting LayouSyn's potential to enable controllable, open-vocabulary scene synthesis without relying on closed-source LLMs.

Abstract

We present Lay-Your-Scene (shorthand LayouSyn), a novel text-to-layout generation pipeline for natural scenes. Prior scene layout generation methods are either closed-vocabulary or use proprietary large language models for open-vocabulary generation, limiting their modeling capabilities and broader applicability in controllable image generation. In this work, we propose to use lightweight open-source language models to obtain scene elements from text prompts and a novel aspect-aware diffusion Transformer architecture trained in an open-vocabulary manner for conditional layout generation. Extensive experiments demonstrate that LayouSyn outperforms existing methods and achieves state-of-the-art performance on challenging spatial and numerical reasoning benchmarks. Additionally, we present two applications of LayouSyn. First, we show that coarse initialization from large language models can be seamlessly combined with our method to achieve better results. Second, we present a pipeline for adding objects to images, demonstrating the potential of LayouSyn in image editing applications.

Figures (13)

• Figure 1: Text to natural scene layout generation with LayouSyn. LayouSyn demonstrates superior scene awareness, generating layouts with high geometric plausibility and strictly adhering to numerical and spatial constraints. Object nouns in the prompts are highlighted with corresponding colors in the layout.
• Figure 2: Overview of inference pipeline for LayouSyn. We frame the scene layout generation task as a two-stage process. First, a lightweight language model extracts a set of relevant object descriptions from the text prompt describing the scene. Second, a trained diffusion model generates layouts conditioned on the text prompt and object descriptions. Note that Concat in the LDiT block refers to the concatenation of description and bounding box tokens along the token dimension, while Sep denotes their separation.
• Figure 3: Diversity of layouts generated by LayouSyn
• Figure 4: Comparative analysis with LayoutGPT. LayouSyn can generate complex layouts with multiple objects following spatial constraints in the prompt.
• Figure 5: Effect of noise schedule scaling on layout fidelity (L-FID) at different CFG scales. A noise schedule scale of 2 and a CFG scale of 2 achieves the lowest L-FID.

Theorems & Definitions (2)

• Theorem 1
• proof