Identifying Memorization of Diffusion Models through p-Laplace Analysis

TL;DR

Using a diffusion-model score, the paper develops a numerical scheme to approximate the $p$-Laplace operator $\Delta_p u$ of the log-density $u=\log p$ and leverages it to detect memorized samples in learned distributions. They propose two formulations, volume and boundary integrals, and show the boundary-based $1$-Laplace estimator is most robust for capturing memorized samples. Through experiments on a Gaussian mixture and a large-scale image-diffusion model (Stable Diffusion v1.4), they demonstrate that memorized prompts produce distinctive low $p$-Laplace values, enabling reliable memorization detection. The results provide a new geometric lens on memorization and a practical diagnostic for privacy-sensitive diffusion-based generation systems.

Abstract

Diffusion models, today's leading image generative models, estimate the score function, i.e. the gradient of the log probability of (perturbed) data samples, without direct access to the underlying probability distribution. This work investigates whether the estimated score function can be leveraged to compute higher-order differentials, namely p-Laplace operators. We show here these operators can be employed to identify memorized training data. We propose a numerical p-Laplace approximation based on the learned score functions, showing its effectiveness in identifying key features of the probability landscape. We analyze the structured case of Gaussian mixture models, and demonstrate the results carry-over to image generative models, where memorization identification based on the p-Laplace operator is performed for the first time.

Figures (3)

• Figure 1: GMM error analysis (take-away: $1$-Laplace boundary formulation is the most reliable). Here we test the fidelity of our approach, to the values of the accessible true probability and its $p$-Laplace. We test learned, and oracle fields, using volume and boundary formulations. Top left: 6 neighborhoods are tested - 3 maxima and 3 non-maxima points. Top middle-right:$\hat{s}$ achieves low error rate in direction estimation, however it is not the case for magnitudes, with relative error of $\sim \times 1.2$ receiving non-negligible frequency. Bottom-right:$1$-Laplace is approximated significantly better than in $p=2,3$ , due to invariance to errors in magnitude. Middle-right: The volume integral formulation has very high variance rendering it less reliable than the boundary formulation, especially for $p=1$. Middle-bottom, left-middle: we provide further per-neighborhood violin plots for $p=1$ showing that, despite the high variance in volume formulation, maxima neighborhoods are still distinguishable from non-maxima.
• Figure 2: We are interested in the ability of the $p$-Laplace of the implicitly learned log probability to reliably reflect memorization, and provide distinguishable values at memorized points. In each row, the scatter-plot shows the GMM training set with randomly drawn peak locations, including one red sample that was replicated 250 times to induce memorization. The colormaps portray the $p$-Laplace, where we see again how the values of the $1$-Laplace pin-point better the memorized sample, assigning it the lowest percentile (percentages in title) compared to the other $p=2,3$-Laplace.
• Figure 3: Testing our approach, using a pre-trained stable diffusion v1.4 to predict the score-function $\hat{s}$. $\hat{s}$ is plugged to the $1$-Laplace boundary integral formulation (Eq. \ref{['eq:boundary_formulation']}) to distinguish memorized from non-memorized samples. Memorized prompt: Mothers influence on her young hippo. Non-memorized prompt is the exact same prompt but with Mother's (with an apostrophe). We generated 100 images with each prompt, and aggregated the criterions to the histogram - which indeed shows good distinguishability. Qualitative examples of 5 generations per prompt are provided below, with their associated $1$-Laplace estimation reported in the title.