Exploring Local Memorization in Diffusion Models via Bright Ending Attention

TL;DR

The paper reframes memorization in diffusion models by introducing a localized memorization perspective and identifying a new phenomenon, bright ending (BE), which reveals how end-token cross-attention concentrates on memorized regions during the final denoising step. BE enables efficient, training-data-free localization of memorized regions, producing automatic local memorization masks that can be integrated into existing evaluation, detection, and mitigation pipelines. By coupling BE with a localized detection metric (LD) and a localized similarity metric (LS), the authors demonstrate improved performance for local memorization tasks while preserving global performance, achieving new state-of-the-art results. Extensive experiments on Stable Diffusion show BE-based localization reduces the gap caused by local memorization and provides a practical tool for policy-relevant privacy and copyright risk assessment in diffusion-model outputs.

Abstract

Text-to-image diffusion models have achieved unprecedented proficiency in generating realistic images. However, their inherent tendency to memorize and replicate training data during inference raises significant concerns, including potential copyright infringement. In response, various methods have been proposed to evaluate, detect, and mitigate memorization. Our analysis reveals that existing approaches significantly underperform in handling local memorization, where only specific image regions are memorized, compared to global memorization, where the entire image is replicated. Also, they cannot locate the local memorization regions, making it hard to investigate locally. To address these, we identify a novel "bright ending" (BE) anomaly in diffusion models prone to memorizing training images. BE refers to a distinct cross-attention pattern observed in text-to-image diffusion models, where memorized image patches exhibit significantly greater attention to the final text token during the last inference step than non-memorized patches. This pattern highlights regions where the generated image replicates training data and enables efficient localization of memorized regions. Equipped with this, we propose a simple yet effective method to integrate BE into existing frameworks, significantly improving their performance by narrowing the performance gap caused by local memorization. Our results not only validate the successful execution of the new localization task but also establish new state-of-the-art performance across all existing tasks, underscoring the significance of the BE phenomenon.

Figures (9)

• Figure 1: Memorization in diffusion models can occur both globally (right) and locally (left). The first row displays the memorized training images, the second row shows the corresponding generated images, and the third row shows the local memorization masks extracted using 'bright ending', where brightness values quantify the memorization effect. Most pixel values in masks on the right are brighter due to global memorization compared to masks on the left due to local memorization.
• Figure 2: For local memorization cases, variations in non-memorized regions significantly impact existing global similarity and detection measures. Left: Using the current similarity metric SSCD, the second row shows memorized images with similarity scores above 0.5, and the third row shows non-memorized images with scores below 0.5. However, the difference between the second and third rows is minimal, differing only in a local portion, and are, in fact, both local memorization cases, which demonstrates the failure of the global SSCD metric. Right: Currently, magnitude is used to detect memorization, where a higher magnitude corresponds to a greater memorization risk. The distribution reveals a noticeable performance gap: local memorization cases are more challenging to detect than global ones, as their magnitude values overlap more with those of non-memorized cases.
• Figure 3: Local memorization tends to have smaller magnitudes than global memorization, making it harder to distinguish from non-memorization. Incorporating BE into the existing detection method effectively increases the magnitude of local cases while keeping the other cases mostly unchanged.
• Figure 4: Visualization of cross-attention maps during the final denoising step in pre-trained Stable Diffusion models. Typically, the end token shows a dark cross-attention map, shifting the denoiser's attention from semantic meanings to fine details. However, in memorized models, 'bright ending' anomaly occurs, where the end token displays abnormally high cross-attention scores, focusing on coarser structures, specifically, the local memorized regions, effectively serving as an efficient automatic extraction of the local memorization mask without needing access to the training data.
• Figure 5: Left: Box plot showing the distribution of end token cross-attention scores during the final inference step for different memorization types. The plot is based on 9,600 generated images, with 16 images generated for 300 memorized and 300 non-memorized prompts each. It clearly distinguishes the three types: non-memorized cases have attention scores close to zero, while memorized cases exhibit abnormally high scores. Global memorization shows higher scores than local memorization due to the larger memorized regions. This further validates the 'bright ending' observation. Right: Violin plots displaying the impact on magnitudes (detection signal) for different memorization types.