Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias

TL;DR

This work identifies a local generation bias in score-based diffusion models as a key source of text-like hallucinations. By introducing the Local Dependency Ratio ($LDR$), the authors quantify how denoising networks rely on localized input regions, often treating symbol generation as nearly independent across tokens. They support this with empirical evidence from synthetic text-like distributions and a theoretical analysis on hypercube-structured data, showing that high $LDR$ emerges early in training and can persist, driving hallucinations even in architectures with global receptive fields. The study also connects these observations to optimization dynamics via a two-layer ReLU model, illustrating how training biases can inhibit learning of global structure, with implications for real-world text and multimodal generation. Overall, the work provides a diagnostic, mechanistic, and theoretical framework for understanding and potentially mitigating hallucinations in diffusion models.

Abstract

Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such as distorted fingers or hallucinated texts with no meaning. This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions of the data distribution are nearly pairwise independent. This behavior leads to a generation process that decomposes the global distribution into separate, independent distributions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising network architectures including MLP and transformers which have the structure to model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as a result of implicit biases in the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving a two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.

Figures (26)

• Figure 1: An illustration for local generation bias. We construct a synthetic dataset (a) that all images satisfy the rule that sum of first row equals second row, i.e. 2+9=7+4. Diffusion model starts from noise $x_t$ (b) and using denoising network to generate digit images in four quarters. We found that the top-left region's denoising primarily depends on its own data, depicted by saliency map (c). This means the diffusion model independently generates each digit without caring any other digits, ends up with $x_0$ (d) failing to capture the relation between four digits.
• Figure 2: Some examples of deformed hands artifacts and text hallucination in images generated by StableDiffusion rombach2022stablediffusion and Midjourney. Images from prompting "woman showing her hands", "a road sign in a grassland" and "a Chinese traditional calligraphy art".
• Figure 3: Experimental Result for UNet learning parity parenthesis $L=16$ (left) and $L=8$ (right).
• Figure 4: Experimental Result for learning Quarter-MNIST using UNet (left) and DiT (right).
• Figure 5: LDR trend for UNet (left) and DiT (right) at different denoising timestep and training iterations. The LDR for UNet remains high throughout the training, therefore it stucks with hallucination. While DiT successfully progress to reduce the LDR, meaning it starts to overfit and memorize the dataset. We select timestep $t$ corresponding to $\sqrt{\bar{\alpha}_t} \approx 0.1,0.3,0.5,0.7,0.9$.

Theorems & Definitions (17)

• Theorem 5.2
• Theorem 5.3
• Lemma A.1
• proof
• Lemma A.2
• proof
• Lemma A.3
• proof
• Theorem A.4
• Lemma A.5