Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models

Ilias Mitsouras, Eleftherios Tsonis, Paraskevi Tzouveli, Athanasios Voulodimos

TL;DR

U-Sketch tackles the challenge of sketch-to-image synthesis with diffusion models by introducing a U-Net latent edge predictor that captures spatial correlations in latent space, complemented by a sketch simplification network. The predictor guides the reverse diffusion process during the initial steps to align outputs with input sketches, while keeping the diffusion backbone pre-trained and inference-time only. Training uses edge supervision from triplets and leverages intermediate denoiser activations to predict latent edges, enabling more realistic and spatially faithful images with far fewer denoising steps than prior per-pixel approaches. User studies and quantitative metrics show that U-Sketch outperforms MLP-based guidance in realism, edge fidelity, and structural coherence, offering substantial efficiency gains and practical impact for sketch-conditioned image generation.

Abstract

Diffusion models have demonstrated remarkable performance in text-to-image synthesis, producing realistic and high resolution images that faithfully adhere to the corresponding text-prompts. Despite their great success, they still fall behind in sketch-to-image synthesis tasks, where in addition to text-prompts, the spatial layout of the generated images has to closely follow the outlines of certain reference sketches. Employing an MLP latent edge predictor to guide the spatial layout of the synthesized image by predicting edge maps at each denoising step has been recently proposed. Despite yielding promising results, the pixel-wise operation of the MLP does not take into account the spatial layout as a whole, and demands numerous denoising iterations to produce satisfactory images, leading to time inefficiency. To this end, we introduce U-Sketch, a framework featuring a U-Net type latent edge predictor, which is capable of efficiently capturing both local and global features, as well as spatial correlations between pixels. Moreover, we propose the addition of a sketch simplification network that offers the user the choice of preprocessing and simplifying input sketches for enhanced outputs. The experimental results, corroborated by user feedback, demonstrate that our proposed U-Net latent edge predictor leads to more realistic results, that are better aligned with the spatial outlines of the reference sketches, while drastically reducing the number of required denoising steps and, consequently, the overall execution time.

U-Sketch: An Efficient Approach for Sketch to Image Diffusion Models

TL;DR

U-Sketch tackles the challenge of sketch-to-image synthesis with diffusion models by introducing a U-Net latent edge predictor that captures spatial correlations in latent space, complemented by a sketch simplification network. The predictor guides the reverse diffusion process during the initial steps to align outputs with input sketches, while keeping the diffusion backbone pre-trained and inference-time only. Training uses edge supervision from triplets and leverages intermediate denoiser activations to predict latent edges, enabling more realistic and spatially faithful images with far fewer denoising steps than prior per-pixel approaches. User studies and quantitative metrics show that U-Sketch outperforms MLP-based guidance in realism, edge fidelity, and structural coherence, offering substantial efficiency gains and practical impact for sketch-conditioned image generation.

Abstract

Diffusion models have demonstrated remarkable performance in text-to-image synthesis, producing realistic and high resolution images that faithfully adhere to the corresponding text-prompts. Despite their great success, they still fall behind in sketch-to-image synthesis tasks, where in addition to text-prompts, the spatial layout of the generated images has to closely follow the outlines of certain reference sketches. Employing an MLP latent edge predictor to guide the spatial layout of the synthesized image by predicting edge maps at each denoising step has been recently proposed. Despite yielding promising results, the pixel-wise operation of the MLP does not take into account the spatial layout as a whole, and demands numerous denoising iterations to produce satisfactory images, leading to time inefficiency. To this end, we introduce U-Sketch, a framework featuring a U-Net type latent edge predictor, which is capable of efficiently capturing both local and global features, as well as spatial correlations between pixels. Moreover, we propose the addition of a sketch simplification network that offers the user the choice of preprocessing and simplifying input sketches for enhanced outputs. The experimental results, corroborated by user feedback, demonstrate that our proposed U-Net latent edge predictor leads to more realistic results, that are better aligned with the spatial outlines of the reference sketches, while drastically reducing the number of required denoising steps and, consequently, the overall execution time.
Paper Structure (13 sections, 5 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 13 sections, 5 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Background
  5. U-Net Latent Edge Predictor
  6. Image Synthesis using U-Net Latent Edge Predictor
  7. Experiments and Results
  8. Experimental Setup
  9. Qualitative Evaluation
  10. Impact of Sketch Simplification Network
  11. Impact of Noise Initialization
  12. Quantitative Evaluation
  13. Conclusion

Figures (6)

  • Figure 1: U-Sketch framework. Given an input sketch $\mathbf{e}$ and a text-prompt $\mathbf{y}$, we start by producing random noise $\mathbf{z}_{t}$ and then we iteratively pass it through the denoising autoencoder ${\boldsymbol{\epsilon}}_{\theta}(\mathbf{z}_{t} | t, \mathbf{y})$ to remove noise. During the first $S$ of the total $T$ denoising steps we guide the synthesis process using our U-Net latent edge predictor and sketch's latent representation $\boldsymbol{\mathcal{E}}(\mathbf{e})$, according to the process described in section \ref{['synthesis_process_section']}. Finally we return the decoded synthesized image $\mathbf{x}_{0} = \boldsymbol{\mathcal{D}}(\mathbf{z}_{0})$. Sketch simplification network $\mathbf{S}$ offers the user the choice of smoothing the input sketch before feeding it in to the pipeline.
  • Figure 2: Sketch-guided text-to-image synthesis examples. For each triplet we have from left to right: the reference sketch along with the text-prompt, the image generated using the MLP and the image generated using our proposed U-Sketch framework.
  • Figure 3: Sketch-guided text-to-image synthesis examples. For each example we have from left to right: the reference sketch along with the text-prompt, the image generated using the MLP with $T = 50$ and $T = 250$ denoising steps and the image generated using our proposed U-Sketch framework with $T = 50$ denoising steps.
  • Figure 4: Sketch-guided text-to-image synthesis examples using U-Sketch. For each triplet we have from left to right: the reference sketch along with the text-prompt, the image generated without the use and with the use of the sketch simplification network.
  • Figure 5: Sketch-guided text-to-image synthesis examples using U-Sketch and different noise initializations.
  • ...and 1 more figures