Label-free Neural Semantic Image Synthesis

TL;DR

This work tackles the challenge of spatially controlling diffusion-based image synthesis without costly annotations by introducing neural semantic image synthesis and the LUMEN framework. It leverages dense features from foundation models to form neural layouts, which undergo PCA-based semantic separation and are used as conditioning in a ControlNet-enabled diffusion pipeline. Empirical results show superior semantic and geometric alignment relative to traditional conditioning (edges, depth, or segmentation) and competitive performance against ground-truth labels in some cases, while preserving diversity. The approach also demonstrates tangible downstream benefits, including synthetic data augmentation for perception tasks and improved cross-domain generalization, highlighting the practicality and scalability of label-free spatial conditioning.

Abstract

Recent work has shown great progress in integrating spatial conditioning to control large, pre-trained text-to-image diffusion models. Despite these advances, existing methods describe the spatial image content using hand-crafted conditioning inputs, which are either semantically ambiguous (e.g., edges) or require expensive manual annotations (e.g., semantic segmentation). To address these limitations, we propose a new label-free way of conditioning diffusion models to enable fine-grained spatial control. We introduce the concept of neural semantic image synthesis, which uses neural layouts extracted from pre-trained foundation models as conditioning. Neural layouts are advantageous as they provide rich descriptions of the desired image, containing both semantics and detailed geometry of the scene. We experimentally show that images synthesized via neural semantic image synthesis achieve similar or superior pixel-level alignment of semantic classes compared to those created using expensive semantic label maps. At the same time, they capture better semantics, instance separation, and object orientation than other label-free conditioning options, such as edges or depth. Moreover, we show that images generated by neural layout conditioning can effectively augment real data for training various perception tasks.

Figures (9)

• Figure 1: Our proposed neural layout conditioning enables the concept of neural semantic image synthesis. Neural layout allows the simultaneous specification of both semantic and spatial concepts, such as scene geometry, object semantics and orientation, all without requiring expensive pixel-wise label annotations for training. This is in contrast to existing conditioning, which as shown here in the red boxes can introduce spatial (Sem.Seg.) or semantic (MiDaS, Canny, HED) ambiguity. Furthermore, neural layouts are compatible with textual prompts, which further enhances diversity in the images generated via neural semantic image synthesis.
• Figure 2: LUMEN uses foundation models (FMs) features to extract neural layouts as conditioning input.
• Figure 3: Visual comparison of four random samples drawn using LUMEN trained with a different number of PCA components. Using fewer trades fidelity for diversity.
• Figure 4: Comparison of images generated with different conditioning types on ADE20k, COCO-Stuff and Cityscapes. Neural layouts provide rich description of the desired images, while other inputs contain limited information and are semantically ambiguous.
• Figure 5: Image manipulation through prompt editing on the COCO-Stuff validation set. We show an unedited sample and additional results from replacing the underlined words in the original caption ($\rightarrow$) or appending additional words at its end ($+$).