Training-Free Sketch-Guided Diffusion with Latent Optimization

TL;DR

This work tackles controllable image synthesis with diffusion models using sketches without any training or fine-tuning. It leverages DDIM inversion to preserve sketch-induced cross-attention maps and introduces latent optimization that aligns intermediate latents with targets derived from the sketch, under a user-provided text prompt. The key insight is that cross-attention maps are robust to domain shifts between sketches and real images and can serve as reliable guidance for structure-aware generation. The method achieves competitive realism and strong layout fidelity on Sketchy and ImageNet-Sketch, and extends to real-image editing with sketch-driven layout/style control, offering a practical, training-free pathway for controllable content creation.

Abstract

Based on recent advanced diffusion models, Text-to-image (T2I) generation models have demonstrated their capabilities to generate diverse and high-quality images. However, leveraging their potential for real-world content creation, particularly in providing users with precise control over the image generation result, poses a significant challenge. In this paper, we propose an innovative training-free pipeline that extends existing text-to-image generation models to incorporate a sketch as an additional condition. To generate new images with a layout and structure closely resembling the input sketch, we find that these core features of a sketch can be tracked with the cross-attention maps of diffusion models. We introduce latent optimization, a method that refines the noisy latent at each intermediate step of the generation process using cross-attention maps to ensure that the generated images adhere closely to the desired structure outlined in the reference sketch. Through latent optimization, our method enhances the accuracy of image generation, offering users greater control and customization options in content creation.

Figures (8)

• Figure 1: Given a sketch and a text prompt, our pipeline synthesizes an image that adheres to the sketch structure and the text description. If the user wants to use an additional exemplar image as another input, we can also perform image variation while maintaining fidelity to the sketch.
• Figure 2: Left: Distributions of inverted sketches (ImageNet-Sketch and Sketchy) show noticeable bias compared to the standard normal distribution and those of inverted real images. Right: Comparison of using different initial noises with different prompts. Using an inverted sketch image generates a sketch-like image even when using the style keyword "photo". Note that the locations of the cats in the generated image align with the highlighted areas in the attention maps.
• Figure 3: Illustration of the proposed latent optimization pipeline and cross-attention visualization. As shown on the right side, we observe that the cross-attention maps remain robust to domain shifts when provided with prompts containing the domain information (see Sec. \ref{['subsec:bridging-gap']}). For image generation, we obtain the inverted sketch noise latents $z_{T}$ through DDIM inversion. Next, we denoise the sketch latents using the source prompt $p_s$ to derive the attention maps corresponding to the sketch image. Finally, we employ randomly sampled initial noise latent $z_{T}^*$ alongside the target prompt $p_t$ to generate a new image. By utilizing KL loss, we facilitate the alignment of the cross-attention maps with those from the sketch.
• Figure 4: Top: Cross-attention maps at time step $t$ from sketch reconstruction (first row) and image generation with random seed $z$ (second row). Bottom: With the same seed $z$, the third row shows attention maps with our optimization, and the fourth row shows the corresponding generated images.
• Figure 5: Sketch to image translations on the Sketchy database sketchy2016 (first row) and the ImageNet-Sketch dataset wang2019learning (second row). Our approach effectively translates these sketch images into realistic images. Even when the sketch is very scribbled, our method can still capture object features in the sketch guide and reproduce them in the generated image.