The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise

TL;DR

This work reveals trigger patches in the initial noise of diffusion models that predictably steer object placement and generalize across prompts. By defining trigger entropy and training a detector, the authors quantify and predict these patches, while two-sample tests show they are outliers in Gaussian noise. They demonstrate applications in increasing location diversity via reject sampling and in improving prompt adherence by injecting trigger patches, achieving a GSR of 83.64% and a detector mAP@50 of 0.325, respectively. Overall, the findings expose a controllable, noise-space mechanism behind diffusion generation with practical implications for editing, diversity, and prompt alignment.

Abstract

Diffusion models have achieved remarkable success in text-to-image generation tasks; however, the role of initial noise has been rarely explored. In this study, we identify specific regions within the initial noise image, termed trigger patches, that play a key role for object generation in the resulting images. Notably, these patches are ``universal'' and can be generalized across various positions, seeds, and prompts. To be specific, extracting these patches from one noise and injecting them into another noise leads to object generation in targeted areas. We identify these patches by analyzing the dispersion of object bounding boxes across generated images, leading to the development of a posterior analysis technique. Furthermore, we create a dataset consisting of Gaussian noises labeled with bounding boxes corresponding to the objects appearing in the generated images and train a detector that identifies these patches from the initial noise. To explain the formation of these patches, we reveal that they are outliers in Gaussian noise, and follow distinct distributions through two-sample tests. Finally, we find the misalignment between prompts and the trigger patch patterns can result in unsuccessful image generations. The study proposes a reject-sampling strategy to obtain optimal noise, aiming to improve prompt adherence and positional diversity in image generation.

Figures (8)

• Figure 1: Illustration: The first row shows images generated from one seed. And we can identify the "trigger patch" located in the red box that tend to induce the generation of the object. If we inject the trigger patch into another noise, there will be objects in the position of the injection place in the images generated by the mixed noise.
• Figure 2: Heatmap of generated objects on two different noises. The left one has a trigger patch while the right one does not.
• Figure 3: Illustration: Each row shows five images generated from one seed. The patch in the bottom row has the strongest effectiveness in inducing the generation of the object, while the objects in the top row are dispersed. So there must exist a strong trigger patch in the orange bounding box in the bottom row noise.
• Figure 4: Trigger Entropy Distribution on the created dataset. Randomly selected bounding boxes as a baseline.
• Figure 5: Object Distribution: For one noise, we plot the scatter to see where the center of the bounding boxes are dispersed.