Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controllable Face Synthesis with Semantic Latent Diffusion Models

Alex Ergasti, Claudio Ferrari, Tomaso Fontanini, Massimo Bertozzi, Andrea Prati

TL;DR

This work introduces an end-to-end Latent Diffusion Model for Semantic Image Synthesis (SIS) of faces that supports both diverse generation under semantic masks and exact reproduction of a reference image's style. By integrating SPADE-based conditioning with a novel mask-conditioned cross-attention mechanism and a Multi-Resolution Style Encoder, the model achieves precise local control over facial regions while maintaining overall image coherence. The approach demonstrates strong reconstruction quality, effective full and partial style transfer, and competitive or superior performance compared to state-of-the-art diffusion methods, with significantly faster inference at practical step counts. These capabilities enable accurate face editing and style customization, offering practical impact for identity-preserving edits, data augmentation, and robust face synthesis research.

Abstract

Semantic Image Synthesis (SIS) is among the most popular and effective techniques in the field of face generation and editing, thanks to its good generation quality and the versatility is brings along. Recent works attempted to go beyond the standard GAN-based framework, and started to explore Diffusion Models (DMs) for this task as these stand out with respect to GANs in terms of both quality and diversity. On the other hand, DMs lack in fine-grained controllability and reproducibility. To address that, in this paper we propose a SIS framework based on a novel Latent Diffusion Model architecture for human face generation and editing that is both able to reproduce and manipulate a real reference image and generate diversity-driven results. The proposed system utilizes both SPADE normalization and cross-attention layers to merge shape and style information and, by doing so, allows for a precise control over each of the semantic parts of the human face. This was not possible with previous methods in the state of the art. Finally, we performed an extensive set of experiments to prove that our model surpasses current state of the art, both qualitatively and quantitatively.

Controllable Face Synthesis with Semantic Latent Diffusion Models

TL;DR

This work introduces an end-to-end Latent Diffusion Model for Semantic Image Synthesis (SIS) of faces that supports both diverse generation under semantic masks and exact reproduction of a reference image's style. By integrating SPADE-based conditioning with a novel mask-conditioned cross-attention mechanism and a Multi-Resolution Style Encoder, the model achieves precise local control over facial regions while maintaining overall image coherence. The approach demonstrates strong reconstruction quality, effective full and partial style transfer, and competitive or superior performance compared to state-of-the-art diffusion methods, with significantly faster inference at practical step counts. These capabilities enable accurate face editing and style customization, offering practical impact for identity-preserving edits, data augmentation, and robust face synthesis research.

Abstract

Semantic Image Synthesis (SIS) is among the most popular and effective techniques in the field of face generation and editing, thanks to its good generation quality and the versatility is brings along. Recent works attempted to go beyond the standard GAN-based framework, and started to explore Diffusion Models (DMs) for this task as these stand out with respect to GANs in terms of both quality and diversity. On the other hand, DMs lack in fine-grained controllability and reproducibility. To address that, in this paper we propose a SIS framework based on a novel Latent Diffusion Model architecture for human face generation and editing that is both able to reproduce and manipulate a real reference image and generate diversity-driven results. The proposed system utilizes both SPADE normalization and cross-attention layers to merge shape and style information and, by doing so, allows for a precise control over each of the semantic parts of the human face. This was not possible with previous methods in the state of the art. Finally, we performed an extensive set of experiments to prove that our model surpasses current state of the art, both qualitatively and quantitatively.
Paper Structure (13 sections, 8 equations, 14 figures, 1 table)

This paper contains 13 sections, 8 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Architecture
  4. Latent Diffusion Model
  5. Multi-Resolution Style Encoder
  6. Mask-conditioned Cross-Attention
  7. SPADE ResBlock
  8. Experiments
  9. Results on reconstruction
  10. Results on total style swap
  11. Results on partial style swap
  12. Results on noise-based Generation
  13. Conclusion

Figures (14)

  • Figure 1: Our model can generate images in three ways: (a) Given a reference image, (b) mixing styles from a reference image and noise, (c) fully noise-based without reference.
  • Figure 2: Model architecture. The encoder part of the UNet uses only standard Resnet Block with SpatialTransformer to guide the diffusion process with the style embedding obtained from $\mathcal{E}_s$. The middle block and the decoder part use SPADEResBlock, as in SDM, to encapsulate the semantic mask info. The Mask attention mechanism is applied inside the SpatialTransformer on the Cross Attention Map.
  • Figure 3: Image reconstruction comparison between the state of the art and our model.
  • Figure 4: Style transfer comparison between different methods and our model. The style of the reference image is applied to the target image. The overall consistency in style swap is far better compared to state-of-the-art methods.
  • Figure 5: Style transfer of single face features. Our model can successfully swap single style part with high image coherence.
  • ...and 9 more figures