PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis

TL;DR

PLACE addresses semantic image synthesis layout fidelity by introducing a Layout Control Map and a timestep-adaptive fusion mechanism to jointly encode layout and semantic information within a pre-trained diffusion model. It augments fine-tuning with a Semantic Alignment loss and a Layout-Free Prior Preservation loss to improve layout fidelity and preserve priors from the pre-trained model. Across ADE20K and COCO-Stuff, PLACE achieves strong in-distribution performance and robust out-of-distribution generalization for new objects, styles, and attributes, with ablations confirming the contribution of each component. The approach provides a practical, high-quality route to controllable semantic image synthesis that maintains pre-trained priors while delivering faithful layouts and realistic details.

Abstract

Recent advancements in large-scale pre-trained text-to-image models have led to remarkable progress in semantic image synthesis. Nevertheless, synthesizing high-quality images with consistent semantics and layout remains a challenge. In this paper, we propose the adaPtive LAyout-semantiC fusion modulE (PLACE) that harnesses pre-trained models to alleviate the aforementioned issues. Specifically, we first employ the layout control map to faithfully represent layouts in the feature space. Subsequently, we combine the layout and semantic features in a timestep-adaptive manner to synthesize images with realistic details. During fine-tuning, we propose the Semantic Alignment (SA) loss to further enhance layout alignment. Additionally, we introduce the Layout-Free Prior Preservation (LFP) loss, which leverages unlabeled data to maintain the priors of pre-trained models, thereby improving the visual quality and semantic consistency of synthesized images. Extensive experiments demonstrate that our approach performs favorably in terms of visual quality, semantic consistency, and layout alignment. The source code and model are available at https://github.com/cszy98/PLACE/tree/main.

Figures (18)

• Figure 1: Overview of our method. (a) We utilize the layout control map calculated from semantic map $S$ and PLACE for layout control. During fine-tuning, we combine the $\mathcal{L}_{LDM}$, $\mathcal{L}_{SA}$, and $\mathcal{L}_{LFP}$ as optimization objective. (b) Calculation of the layout control map and details of the adaptive layout-semantic fusion module. Each vector in the Layout Control Map encodes all the semantic components in the reception field. The adaptive layout-semantic fusion module blends the layout and semantics feature in a timestep-adaptive way.
• Figure 2: Comparison between naive resize and layout control map regarding information preservation (downsampling by 8 times). The $1st$ column displays the original mask of 'plants, lights' and full semantic map, the $2nd$ column shows the nearest resized mask and corresponding synthesized image, and the $3rd$ column presents the representation of the layout control map and its generated image. A higher value indicates a higher proportion of semantics within its patch. Ours preserves more details.
• Figure 3: Analysis of adaptive fusion module. (a) shows the variation of adaptive $\alpha$ with respect to the timestep. The $\alpha$ decreases as the timestep progresses. (b) presents the layout control map of the 'sidewalk' and the corresponding comparison of the fusion maps (at $t=800/1000$) between fixed $\alpha=1$ and adaptive $\alpha$. (c) illustrates the variation of predicted $\hat{x}_0$ with respect to the sampling steps: one with a fixed $\alpha$ and the other with an adaptive $\alpha$. The latter leads to the synthesis of more realistic details. Zoom in for details.
• Figure 4: Calculation of the Semantic Alignment loss.
• Figure 5: Visual comparisons on ADE20K ($1st \sim 3rd$ rows) and COCO-Stuff ($4th \sim 5th$ rows).