SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation

TL;DR

A novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance that achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works.

Abstract

Despite significant progress in Text-to-Image (T2I) generative models, even lengthy and complex text descriptions still struggle to convey detailed controls. In contrast, Layout-to-Image (L2I) generation, aiming to generate realistic and complex scene images from user-specified layouts, has risen to prominence. However, existing methods transform layout information into tokens or RGB images for conditional control in the generative process, leading to insufficient spatial and semantic controllability of individual instances. To address these limitations, we propose a novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance. Owing to rich spatial and semantic information encapsulated in well-designed feature maps, SSMG achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works. Additionally, we propose the Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA) mechanisms. The former aims to model the relationships among multiple objects within scenes while the latter is designed to heighten the model's sensitivity to the spatial information embedded in the guidance. Extensive experiments demonstrate that SSMG achieves highly promising results, setting a new state-of-the-art across a range of metrics encompassing fidelity, diversity, and controllability.

Figures (6)

• Figure 1: A comparison of image generation methods with different guidance. In contrast to the prevalent token-guided or image-guided methods, our map-guided method excels in providing superior control over both the spatial arrangement and semantic details of individual instances, thereby leading to higher-quality and more appropriate results.
• Figure 2: The overall architecture of the proposed SSMG. During the conditional generation process, we first leverage the VQ-GAN's latent encoder within Stable Diffusion (SD) to downsize the entire dataset of 512 × 512 images into the 64 × 64 latent space. To ensure consistency, we also transform the spatial-semantic map to the 64 × 64 latent space in the condition branch. Subsequently, we duplicate the structures and weights of the SD encoder and middle block as ControlNet. The latent map is then fed into the copied SD encoder and middle block to produce conditional feature maps at different scales. These conditional feature maps are then integrated with the corresponding blocks in the SD decoder and middle block through the proposed LSA.
• Figure 3: Illustrations of the RSA and LSA mechanisms. Feature maps are flattened before being fed into attention.
• Figure 4: Qualitative comparison with SOTA methods. The red boxes indicate unrecognizable or mispositioned instances.
• Figure 5: Illustrations of free-form textual descriptions.