On Memorization in Diffusion Models

TL;DR

This paper addresses the memorization gap in diffusion models by formalizing effective model memorization (EMM) and conducting a large-scale empirical study across data distributions, model configurations, and training procedures. It demonstrates that the theoretical optimum for diffusion models memorizes training data, while learned models can generalize on large datasets but memorize on smaller ones; conditioning on random or unique labels dramatically increases memorization. The work identifies key factors—data dimension, model width, time-embedding choice, and high-resolution skip connections—that modulate EMM, and provides practical insights for users and theoretical directions for deep generative modeling. The released code and findings have implications for privacy, copyright, and responsible use of diffusion-based generators.

Abstract

Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.e., denoising score matching, has a closed-form optimal solution that can only generate training data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of state-of-the-art diffusion models, and thus calls for a deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates our definition of effective model memorization (EMM), a metric measuring the maximum size of training data at which a learned diffusion model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of EMM, focusing primarily on data distribution, model configuration, and training procedure. Besides comprehensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models. Code is available at https://github.com/sail-sg/DiffMemorize.

Figures (22)

• Figure 1: Overall motivation. Generated images (top row) and their $\ell_2$-nearest training samples in $\mathcal{D}$ (bottom row) by (a) the theoretical optimum defined in equation \ref{['optimal']}; (b) EDM karras2022elucidating. Memorization Ratios (%) of EDM models trained with different $|\mathcal{D}|$; (c) within 4k training epochs; (d) when extending to 40k training epochs.
• Figure 2: Memorization ratios (%) of diffusion models on CIFAR-10 under different factors of data distribution $\mathbf{P}$. The intersections of dashed line (90%) and different curves are the estimates of EMMs.
• Figure 3: Memorization ratios (%) of diffusion models on CIFAR-10 under different factors of model configuration $\mathcal{M}$.
• Figure 4: Memorization ratio (%) of diffusion models on CIFAR-10 when retaining (a) skip connections of certain spatial resolution for DDPM++; (b) single skip connection at different locations for DDPM++; (c) skip connections of certain spatial resolution for NCSN++; (d) single skip connection at different locations for NCSN++.
• Figure 5: Memorization ratios (%) of (a) unconditional diffusion models and conditional diffusion models on CIFAR-10 with true / random labels; (b) conditional diffusion models with $C$ random labels; (c) conditional EDM with unique labels and unconditional EDM at $|\mathcal{D}|=\,$50k during the training.

Theorems & Definitions (1)

• Definition 2.1: Effective model memorization