Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model

TL;DR

This work examines the intermediate statuses during the gradual denoising generation process in DPM and concludes that in the earlier generation stage, the image is mostly decided by the special token [\texttt{EOS}] in the text prompt, and the information in the text prompt is already conveyed in this stage.

Abstract

Recently, the strong latent Diffusion Probabilistic Model (DPM) has been applied to high-quality Text-to-Image (T2I) generation (e.g., Stable Diffusion), by injecting the encoded target text prompt into the gradually denoised diffusion image generator. Despite the success of DPM in practice, the mechanism behind it remains to be explored. To fill this blank, we begin by examining the intermediate statuses during the gradual denoising generation process in DPM. The empirical observations indicate, the shape of image is reconstructed after the first few denoising steps, and then the image is filled with details (e.g., texture). The phenomenon is because the low-frequency signal (shape relevant) of the noisy image is not corrupted until the final stage in the forward process (initial stage of generation) of adding noise in DPM. Inspired by the observations, we proceed to explore the influence of each token in the text prompt during the two stages. After a series of experiments of T2I generations conditioned on a set of text prompts. We conclude that in the earlier generation stage, the image is mostly decided by the special token [\texttt{EOS}] in the text prompt, and the information in the text prompt is already conveyed in this stage. After that, the diffusion model completes the details of generated images by information from themselves. Finally, we propose to apply this observation to accelerate the process of T2I generation by properly removing text guidance, which finally accelerates the sampling up to 25\%+.

Figures (22)

• Figure 1: Figure \ref{['fig:cross-attention map']} is the averaged cross-attention over denoising steps. The two generated images are on the top, and the weights in cross-attention maps of each tokens are on the bottom with whiter pixels correspond to larger weights in cross-attention map. Figure \ref{['fig:cross-attention map ratio']} is obtained by taking average over tokens and prompts in PromptSet, which compares the shapes of cross-attention map and final generated images, Measured by relative F1-score $\text{F1}_{t} / \text{F1}_{1}$ over different denoising steps.
• Figure 1: The alignment of generated image with its source and target prompts. The prompts are constructed with switched [EOS].
• Figure 2: Figure \ref{['fig:corrupted data']} visualizes the completed noisy data and its high-frequency, and low-frequency parts over different time steps, listed from top to bottom. Figures \ref{['fig:norm']} and \ref{['fig:ratio']} measure the low/high-frequency signals of $\boldsymbol{x}_{t}$. In Figure \ref{['fig:norm']}, "Low_Add_Noisy_Data/eps" means the norm of $\sqrt{\bar{\alpha}_{t}}\boldsymbol{x}_{0}^{\rm low}$ and $\sqrt{1 - \bar{\alpha}_{t}}\boldsymbol{\epsilon}_{t}^{\rm low}$, vise versa for "High_...". On the other hand, Figure \ref{['fig:ratio']} measures the variation ratio of high/low frequency parts of images during the noising/denoising process. For example, "High_Add_Noise" represents $\|\boldsymbol{x}_{t}^{\rm high} - \boldsymbol{x}_{0}^{\rm high}\| / \|\boldsymbol{x}_{0}^{\rm high}\|$ during noising process.
• Figure 3: Averaged weights in cross-attention map over pixels of three classes of tokens. For each prompt in PromptSet, the result is obtained by taking average over tokens in each class. The final result is the average over PromptSet. Notably, the weights on [SOS] are all larger than 0.9.
• Figure 4: Images under prompts from S-PromptSet with switched [EOS]. The objects are consistent with the ones conveyed by [EOS], while some information in semantic tokens is still conveyed.

Theorems & Definitions (5)

• Proposition 1
• Remark 1
• Remark 2
• Proposition 1
• proof