Enhancing Privacy-Utility Trade-offs to Mitigate Memorization in Diffusion Models

TL;DR

Memorization in text-to-image diffusion models creates privacy and copyright risks as outputs can echo training images. PRSS merges prompt re-anchoring (PR) and semantic prompt search (SS) to refine classifier-free guidance during inference, reducing memorization while preserving user intent, all without retraining. Empirical results show PR and SS provide complementary gains—PR improves privacy with modest utility cost, while SS boosts utility with limited privacy impact—yielding state-of-the-art privacy-utility trade-offs across privacy levels, especially for global memorization. The approach is lightweight to implement, requiring only CFG updates and LLM-based prompt diversification, making it practical for deployment and adaptable to future detection signals. Overall, PRSS significantly advances practical memorization mitigation in diffusion models while maintaining alignment with user prompts.

Abstract

Text-to-image diffusion models have demonstrated remarkable capabilities in creating images highly aligned with user prompts, yet their proclivity for memorizing training set images has sparked concerns about the originality of the generated images and privacy issues, potentially leading to legal complications for both model owners and users, particularly when the memorized images contain proprietary content. Although methods to mitigate these issues have been suggested, enhancing privacy often results in a significant decrease in the utility of the outputs, as indicated by text-alignment scores. To bridge the research gap, we introduce a novel method, PRSS, which refines the classifier-free guidance approach in diffusion models by integrating prompt re-anchoring (PR) to improve privacy and incorporating semantic prompt search (SS) to enhance utility. Extensive experiments across various privacy levels demonstrate that our approach consistently improves the privacy-utility trade-off, establishing a new state-of-the-art.

Figures (13)

• Figure 1: Examples of Local Memorization: Stable Diffusion generations (columns 2–4) can replicate local regions of training images (column 1). Our PRSS method effectively mitigates memorization while maintaining strong alignment with the input prompts (columns 5-7).
• Figure 2: Examples of Global Memorization: Stable Diffusion generations (columns 2–4) can replicate the entire training images (column 1). Our PRSS method effectively mitigates memorization while maintaining strong alignment with the input prompts (columns 5-7).
• Figure 3: Illustration of the logic behind how $\lambda$ in \ref{['eq:baseline']} governs the privacy-utility trade-off. Currently, the privacy-utility trade-off is governed solely by how extensively the prompt is modified.
• Figure 4: Illustration of our proposed strategy (PRSS) compared to the baseline prompt engineering (PE) method. a) The privacy-utility trade-off in the baseline method, where the privacy increases at the cost of utility from $\textcolor{blue}{B}$ to $\textcolor{blue}{C}$ to $\textcolor{blue}{D}$. b) Our proposed PR guidance, i.e, the green arrow, drives the image out of the low-privacy region more efficiently than the baseline guidance, i.e, the black arrows. c) Our proposed SS can enhance utility, by finding points $\textcolor{orange}{B'C'D'}$ that are at the same level of privacy as $\textcolor{blue}{BCD}$ but with better utility. In summary, using the proposed PR and SS jointly can enhance the privacy-utility trade-off to mitigate memorization in diffusion models.
• Figure 5: Both the baseline approach and our method effectively inhibit Stable Diffusion from producing memorized images by minimizing the magnitude in the initial inference step for prompt "The No Limits Business Woman Podcast". Nonetheless, our semantic prompt search (SS) strategy, which results in "The Empowered Business Woman’s Podcast", retains considerably more utility, as demonstrated by the substantially higher CLIP similarities between the original user text prompt and both our (1) modified prompt, and (2) ultimately generated image. Qualitative results further corroborate this finding. $m_{T-1}$ is the first-step magnitude, $CLIP_{txt}$ and $CLIP_{img}$ refer to the original prompt's CLIP similarity with the modified prompt and the generated image.