A Geometric Framework for Understanding Memorization in Generative Models

TL;DR

The paper proposes the manifold memorization hypothesis (MMH), a geometric framework that explains memorization in deep generative models through the lens of local intrinsic dimension (LID) on the ground-truth data manifold $\\mathcal{M}_*$ and the model manifold $\\mathcal{M}_\\theta$. Memorization occurs at points where $x$ lies on $\\mathcal{M}_\\theta$ but has insufficient local dimensionality, captured by the relation $\\text{LID}_\\theta(x) < \\text{LID}_*(x)$, and is categorized into overfitting-driven (OD-Mem) and data-driven (DD-Mem) memorization. The authors validate MMH across toy data and real-world image models, develop and compare practical LID estimators (FLIPD, NB, Local PCA), and demonstrate mitigation strategies that increase LID during sampling (including CFG-based and token-attribution approaches). They also connect MMH to existing literature, offering a unified explanation for duplications, reconstructive memorization, and conditioning-induced memorization, while acknowledging limitations in estimator overlap and the need for more robust LID tools. Overall, MMH provides actionable diagnostics and mitigations for memorization, with significant implications for privacy, copyright risk, and safer deployment of generative systems. The work demonstrates that guiding generated samples toward higher local intrinsic dimensionality can reduce memorized outputs, offering a practical pathway to balance memorization risks with generation quality in large-scale diffusion models.

Abstract

As deep generative models have progressed, recent work has shown them to be capable of memorizing and reproducing training datapoints when deployed. These findings call into question the usability of generative models, especially in light of the legal and privacy risks brought about by memorization. To better understand this phenomenon, we propose the manifold memorization hypothesis (MMH), a geometric framework which leverages the manifold hypothesis into a clear language in which to reason about memorization. We propose to analyze memorization in terms of the relationship between the dimensionalities of (i) the ground truth data manifold and (ii) the manifold learned by the model. This framework provides a formal standard for "how memorized" a datapoint is and systematically categorizes memorized data into two types: memorization driven by overfitting and memorization driven by the underlying data distribution. By analyzing prior work in the context of the MMH, we explain and unify assorted observations in the literature. We empirically validate the MMH using synthetic data and image datasets up to the scale of Stable Diffusion, developing new tools for detecting and preventing generation of memorized samples in the process.

Figures (18)

• Figure 1: An illustrative example of $\text{LID}$ values for models with different quality of fit and degrees of memorization. In these plots, the ground truth manifold $\mathcal{M}_*$ is depicted in light blue, training samples $\{x_i\}_{i=1}^n \subset \mathcal{M}_*$ are depicted as crosses, and the model manifolds $\mathcal{M}_\theta$ are depicted in red. In (a) and (d), the model assigns 0-dimensional point masses around the three leftmost datapoints, indicating that it will reproduce them directly at test time; however in the former case this is caused by overfitting ($\text{LID}_\theta(x)<\text{LID}_*(x)$), while in the latter case it is caused by the ground truth data having small LID. The models in (b) and (e) are analoguous to (a) and (b), respectively, and still memorize, but with an extra degree of freedom in the form of a 1-dimensional submanifold containing the three points. Only the model in (c), which has learned a 2-dimensional manifold through its full support, has generalized well enough and has learned a manifold of high enough dimension to avoid both types of memorization. Finally, (f) shows a poorly fit model where LID and memorization are not meaningfully related.
• Figure 2: 8 images along a relatively low-dimensional manifold learned by Stable Diffusion v1.5. The first is a real image from LAION (flagged as memorized by webster2023reproducible), and the remainder were generated by the model.
• Figure 3: CFG-adjusted scores vs CFG vectors for Stable Diffusion with $\lambda=7.5$ and $t=0.02$ on $20$ memorized and $20$ non-memorized images from LAION.
• Figure 4: Training a diffusion model on a von Mises mixture. (Top) Ground truth manifold and the associated distribution. (Bottom) Model-generated samples with their LID estimates.
• Figure 5: Visualizing OD-Mem and DD-Mem on StyleGAN2-ADA and iDDPM trained on CIFAR10.

Theorems & Definitions (19)

• Proposition 3.0: Informal
• proof
• Proposition 3.0: Informal
• proof
• Definition A.0
• Proposition A.0
• proof
• Theorem A.1: Informal
• proof
• Lemma E.0