Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability

Rohan Asthana, Vasileios Belagiannis

TL;DR

Memorization in diffusion models is addressed by showing norm-based detectors fail in the anisotropic low-noise regime. The authors propose a denoising-free metric that combines isotropic score-norm with anisotropic angular alignment, computed from two forward passes on pure-noise inputs at $t\approx 0$ and $t\approx T$. They provide a theoretical bound on the cosine similarity between unconditional and conditional scores and empirically demonstrate superior detection (AUC and $\mathrm{TPR}_{1\%\mathrm{FPR}}$) on Stable Diffusion $v1.4$ and $v2.0$, with at least a 5x speedup over prior methods; they also validate an inference-time mitigation via prompt augmentation that yields non-memorized, text-aligned outputs. The approach generalizes to Realistic Vision and offers practical benefits for privacy-aware generation and safe deployment of diffusion models.

Abstract

Diffusion-based image generative models produce high-fidelity images through iterative denoising but remain vulnerable to memorization, where they unintentionally reproduce exact copies or parts of training images. Recent memorization detection methods are primarily based on the norm of score difference as indicators of memorization. We prove that such norm-based metrics are mainly effective under the assumption of isotropic log-probability distributions, which generally holds at high or medium noise levels. In contrast, analyzing the anisotropic regime reveals that memorized samples exhibit strong angular alignment between the guidance vector and unconditional scores in the low-noise setting. Through these insights, we develop a memorization detection metric by integrating isotropic norm and anisotropic alignment. Our detection metric can be computed directly on pure noise inputs via two conditional and unconditional forward passes, eliminating the need for costly denoising steps. Detection experiments on Stable Diffusion v1.4 and v2 show that our metric outperforms existing denoising-free detection methods while being at least approximately 5x faster than the previous best approach. Finally, we demonstrate the effectiveness of our approach by utilizing a mitigation strategy that adapts memorized prompts based on our developed metric.

Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability

TL;DR

Memorization in diffusion models is addressed by showing norm-based detectors fail in the anisotropic low-noise regime. The authors propose a denoising-free metric that combines isotropic score-norm with anisotropic angular alignment, computed from two forward passes on pure-noise inputs at and . They provide a theoretical bound on the cosine similarity between unconditional and conditional scores and empirically demonstrate superior detection (AUC and ) on Stable Diffusion and , with at least a 5x speedup over prior methods; they also validate an inference-time mitigation via prompt augmentation that yields non-memorized, text-aligned outputs. The approach generalizes to Realistic Vision and offers practical benefits for privacy-aware generation and safe deployment of diffusion models.

Abstract

Diffusion-based image generative models produce high-fidelity images through iterative denoising but remain vulnerable to memorization, where they unintentionally reproduce exact copies or parts of training images. Recent memorization detection methods are primarily based on the norm of score difference as indicators of memorization. We prove that such norm-based metrics are mainly effective under the assumption of isotropic log-probability distributions, which generally holds at high or medium noise levels. In contrast, analyzing the anisotropic regime reveals that memorized samples exhibit strong angular alignment between the guidance vector and unconditional scores in the low-noise setting. Through these insights, we develop a memorization detection metric by integrating isotropic norm and anisotropic alignment. Our detection metric can be computed directly on pure noise inputs via two conditional and unconditional forward passes, eliminating the need for costly denoising steps. Detection experiments on Stable Diffusion v1.4 and v2 show that our metric outperforms existing denoising-free detection methods while being at least approximately 5x faster than the previous best approach. Finally, we demonstrate the effectiveness of our approach by utilizing a mitigation strategy that adapts memorized prompts based on our developed metric.
Paper Structure (37 sections, 1 theorem, 26 equations, 6 figures, 8 tables)

This paper contains 37 sections, 1 theorem, 26 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memorization in Diffusion Models
  4. Low-noise regime of Diffusion Models
  5. Preliminaries
  6. Score-based Diffusion Models
  7. Norm-based Memorization Detection
  8. Method
  9. Anisotropy in Log-Probability
  10. Anisotropy in Low-Noise Regime
  11. Failure of Norm-Based Methods in Anisotropy
  12. Memorization through Angular Alignment
  13. Detection Metric and Mitigation
  14. Detection
  15. Inference-time Mitigation
  16. ...and 22 more sections

Key Result

Theorem 1

Consider the anisotropic low-noise regime of diffusion and let $\Sigma_t,\Sigma^c_t$ be symmetric positive definite and $v_t := \mathbf{x}_t-\mu$ and $\delta := \mu_c-\mu$ denote the sample displacement from the unconditional mode and the relative displacement between the guidance mode and unconditi Let $\alpha>0$ and constants $\varepsilon,\tau \ge 0$. Assume and set $r:=\varepsilon+\tau<1$. The

Figures (6)

  • Figure 1: Variance of eigenvalues of the Hessian during denoising.
  • Figure 2: Histograms and Kernel Density Estimation (KDE) curves of Wen's norm-based metric $\|s_\theta^\Delta(\mathbf{x}_t, t ,c)\|$wen2024detecting in isotropy ($t\approx T$) and anisotropy ($t\approx0$). We observe a larger overlap of KDE curves in anisotropy compared to isotropy, which indicates poor discrimination capabilities between memorized and non-memorized samples.
  • Figure 3: Comparison of angular alignment between $\nabla_{\mathbf{x}_t}\log p_t(\mathbf{x}_t)$ and $\nabla_{\mathbf{x}_t}\log p_t(c | \mathbf{x}_t)$, along with the heatmap of cosine similarity between them, for memorized and non-memorized cases. (a) We observe a larger number of highly-aligned vectors for the memorized case compared to the non-memorized case, indicated through orange rings. (b) We observe generally a higher cosine similarity (indicated as red regions) in the memorized case compared to the non-memorized case.
  • Figure 4: (a,b) Quantitative comparison of inference-time mitigation methods on SD v1.4 and SD v2.0. The evaluation is done across five distinct hyperparameter configurations. Lower values of SSCD Similarity and higher values of CLIP Score and Aesthetic Score are desirable. (c) Qualitative comparison of inference-time mitigation strategies on SD v1.4.
  • Figure 5: Qualitative comparison of inference-time mitigation approaches on SD v1.4. Specifically, we visualize the memorized training image (left-most), along with the mitigated generated images by the methods from ren2024unveiling, wen2024detecting, jeon2025understanding, and our method (right-most).
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Remark 1