MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

TL;DR

MemBench tackles the memorization risk in diffusion-based text-to-image generation by providing a large, benchmarked dataset of memorized image trigger prompts and a two-scenario evaluation framework that includes both trigger prompts and general prompts. It introduces an MCMC-based method guided by a $D_{\bm{\theta}}$ objective to efficiently discover memorized prompts without direct access to training data, and constructs the dataset in two stages using masked priors and prior prompts, complemented by verification via reverse image search and human checks. The study demonstrates that existing memorization mitigation methods degrade text-image alignment and image quality, and often harm general-prompt performance, underscoring the need for robust benchmarks like MemBench. By delivering rigorous metrics, reference performance, and an open-source pipeline, MemBench paves the way for more reliable evaluation and future improvements in memorization mitigation for diffusion models.

Abstract

Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often generate replicated images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, there have been recent studies for developing memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks impedes the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in various Text-to-Image diffusion models. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. This is an important development considering the practical applications which previous works have overlooked. Through evaluation on MemBench, we verify that the performance of existing image memorization mitigation methods is still insufficient for application to diffusion models. The code and datasets are available at https://github.com/chunsanHong/MemBench\_code.

Figures (8)

• Figure 1: Components of Memorized Images in Stable Diffusion 1
• Figure 2: The necessity of measuring the Aesthetic score. Images generated with the mitigation method applied are not desirable but achieve a low SSCD while maintaining a high CLIP Score.
• Figure 4: Results of images found by leveraging Reverse Image Search API to the images generated from trigger prompts. The shared layout suggests the occurrence of image memorization.
• Figure : (a)
• Figure : (a)

Theorems & Definitions (1)

• Definition 1: $\tau$-Image Memorization