Finding Dori: Memorization in Text-to-Image Diffusion Models Is Not Local

TL;DR

This work interrogates the locality assumption behind memorization mitigation in text-to-image diffusion models, showing that pruning-inspired approaches like NeMo and Wanda fail to permanently erase verbatim memorization, as adversarial embeddings can still trigger replication. By crafting Dori, the authors demonstrate that memorization triggers are distributed across the text embedding space and that memorization can be reactivated via diverse activations, undermining locality-based strategies. They then propose a global mitigation via adversarial fine-tuning that updates all model parameters while using surrogate images to preserve utility, achieving robust removal of memorization against adversarial triggers. The findings highlight the need for model-wide memorization defenses beyond prompt- or weight-locality assumptions and point to practical pathways for safer deployment of diffusion models. Overall, the paper contributes a rigorous evaluation of memorization locality, introduces a powerful adversarial toolkit for probing memory, and demonstrates a robust mitigation paradigm with potential to inform future safety mechanisms in generative AI.

Abstract

Text-to-image diffusion models (DMs) have achieved remarkable success in image generation. However, concerns about data privacy and intellectual property remain due to their potential to inadvertently memorize and replicate training data. Recent mitigation efforts have focused on identifying and pruning weights responsible for triggering verbatim training data replication, based on the assumption that memorization can be localized. We challenge this assumption and demonstrate that, even after such pruning, small perturbations to the text embeddings of previously mitigated prompts can re-trigger data replication, revealing the fragility of such defenses. Our further analysis then provides multiple indications that memorization is indeed not inherently local: (1) replication triggers for memorized images are distributed throughout text embedding space; (2) embeddings yielding the same replicated image produce divergent model activations; and (3) different pruning methods identify inconsistent sets of memorization-related weights for the same image. Finally, we show that bypassing the locality assumption enables more robust mitigation through adversarial fine-tuning. These findings provide new insights into the nature of memorization in text-to-image DMs and inform the development of more reliable mitigations against DM memorization.

Figures (20)

• Figure 1: Left:Without mitigation, the DM closely replicates the training sample. Mitigation strategies, such as pruning memorization neurons with NeMo hintersdorf2024finding or Wanda chavhan2024memorization, prevent replication for the memorized prompt, thereby suggesting successful removal. Yet, adversarial embeddingsstill trigger replication. Right: While pruning alters the generation trajectory for the original memorized prompt (blue), adversarial embeddings steer denoising along alternative paths (red) that still lead to the memorized content, unaffected by the pruning-based mitigation.
• Figure 2: Data replication triggers are widely and uniformly scattered in the text embedding space.
• Figure 3: Diverse activations refute locality. Although adversarial embeddings trigger the same image, their activations exhibit high discrepancy.
• Figure 4: Locality fails in the model's weights. Large activation discrepancy (\ref{['fig:mem_not_local_acts']}) results in low weight agreement, further undermining the idea that weights responsible for replicating a memorized image can be pinpointed and pruned.
• Figure 5: Arbitrary image replication. We find that when pushed to the extreme, Dori search yields generations (columns from two to six from the left) of non-memorized data (first from the left).