Enhancing Image Layout Control with Loss-Guided Diffusion Models

TL;DR

This work provides an interpretation for theseusion models which highlights their complimentary features, and demonstrates that it is possible to obtain superior performance when both methods are used in concert.

Abstract

Diffusion models are a powerful class of generative models capable of producing high-quality images from pure noise using a simple text prompt. While most methods which introduce additional spatial constraints into the generated images (e.g., bounding boxes) require fine-tuning, a smaller and more recent subset of these methods take advantage of the models' attention mechanism, and are training-free. These methods generally fall into one of two categories. The first entails modifying the cross-attention maps of specific tokens directly to enhance the signal in certain regions of the image. The second works by defining a loss function over the cross-attention maps, and using the gradient of this loss to guide the latent. While previous work explores these as alternative strategies, we provide an interpretation for these methods which highlights their complimentary features, and demonstrate that it is possible to obtain superior performance when both methods are used in concert.

Figures (7)

• Figure 1: The effects of varying the strengths of BoxDiff and attention injection by tuning their respective parameters. In the top row, we sweep through various choices of $\alpha_T$ to tune the guidance strength of BoxDiff. In the bottom row, we sweep through various choices of the injection strength $\nu'$. For iLGD, shown in the final column, we use $\nu'=0.75$ and $\eta = 0.8$.
• Figure 1: A comparison of iLGD against BoxDiff, Chen et al., MultiDiffusion, and Stable Diffusion, using the same prompts as Figure 3 but different random seeds, with the seed kept the same across each set of images.
• Figure 2: Images generated with the prompt "a ball on the grass," using the bounding boxes shown in the first column. Each row corresponds to a different method. The bounding boxes in the first column are used for injection and iLGD. The attention maps in the second column are averages over the $8\times8$ resolution attention maps at $t=0$ over 100 random seeds. Each of the 8 columns of images in this figure corresponds to one of these 100 seeds.
• Figure 2: A comparison of iLGD against BoxDiff, Chen et al., MultiDiffusion, and Stable Diffusion. The random seed kept the same across each set of images.
• Figure 3: A graphical depiction of injection loss guidance (iLGD).