Laplacian Score Sharpening for Mitigating Hallucination in Diffusion Models

TL;DR

This work addresses hallucinations in diffusion models arising from mode interpolation and score smoothing by introducing a post-hoc score sharpening technique that leverages the score's Laplacian to boost uncertain inter-mode regions during sampling. The method estimates the high-dimensional Laplacian efficiently via a finite-difference Hutchinson trace estimator, enabling application to images, and demonstrates substantial reductions in hallucinations on 1D/2D synthetic data and a Shapes dataset, while largely preserving overall distribution fidelity. Key contributions include the formulation of peak sharpening with $R_j = Y_j - \alpha Y_j''$, the Hutchinson-based Laplacian estimator for high dimensions, and empirical evidence linking Laplacian magnitude to hallucination dynamics during sampling. While effective, the approach is somewhat destructive and hyperparameter-sensitive, motivating future work on adaptive integration into the sampling process and automated hyperparameter tuning to improve practicality.

Abstract

Diffusion models, though successful, are known to suffer from hallucinations that create incoherent or unrealistic samples. Recent works have attributed this to the phenomenon of mode interpolation and score smoothening, but they lack a method to prevent their generation during sampling. In this paper, we propose a post-hoc adjustment to the score function during inference that leverages the Laplacian (or sharpness) of the score to reduce mode interpolation hallucination in unconditional diffusion models across 1D, 2D, and high-dimensional image data. We derive an efficient Laplacian approximation for higher dimensions using a finite-difference variant of the Hutchinson trace estimator. We show that this correction significantly reduces the rate of hallucinated samples across toy 1D/2D distributions and a high-dimensional image dataset. Furthermore, our analysis explores the relationship between the Laplacian and uncertainty in the score.

Figures (7)

• Figure 1: Illustration of the mode interpolation phenomenon in diffusion models, generated samples fall in between the true data modes, resulting in hallucinations
• Figure 2: (a) A toy 1D distribution with modes at $x=\{1,2,3\}$ along with its true score function (blue) and the second derivative of the true score. (b) A visualization of score sharpening in the 1D scenario, from top to bottom: the learned score function (orange) corresponding to the distribution in (a), the second derivative (Laplacian) of the learned score, and the new sharpened score function (green).
• Figure 3: Histograms of the true 1D distribution (blue) and generated distributions using the vanilla (orange) and sharpened (green) scores for 100,000 samples. Sharpening the score reduces the number of samples generated in inter-mode regions.
• Figure 4: A Scatter Plot of the true 2D data distribution for 100,000 samples is shown in blue. Samples generated by the vanilla model (orange) exhibit grid-like interpolation artifacts between modes, which are reduced in samples generated using our sharpened score (red).
• Figure 5: (a) Samples on the true data manifold of Shapes Dataset, (b) Samples generated by the diffusion model, which are not part of the true data manifold.