Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection

Abstract

State-of-the-art text-to-image diffusion models can produce impressive visuals but may memorize and reproduce training images, creating copyright and privacy risks. Existing prompt perturbations applied at inference time, such as random token insertion or embedding noise, may lower copying but often harm image-prompt alignment and overall fidelity. To address this, we introduce two complementary methods. First, Region-Aware Prompt Augmentation (RAPTA) uses an object detector to find salient regions and turn them into semantically grounded prompt variants, which are randomly sampled during training to increase diversity, while maintaining semantic alignment. Second, Attention-Driven Multimodal Copy Detection (ADMCD) aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation, and applies simple thresholded decision rules to detect copying without training with large annotated datasets. Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.

Figures (7)

• Figure 1: Overview of our framework.(a) RAPTA augments the training caption in an image-grounded way: a detector provides region proposals with classes and coarse positions; a small template set instantiates region-aware variants from the base prompt. One variant is sampled per iteration to condition the conditional diffusion model, injecting grounded diversity while preserving prompt–image alignment. (b) ADMCD is an inference-time, two-step detector. For a generated image $G$, we compare it against the training set using fused-feature cosine similarity $S_{\mathrm{fus}}$; pairs with $S_{\mathrm{fus}}>\tau_1$ are flagged as Copy. A similarity-weighted score over three streams (visual, CLIP, texture) with threshold $\tau_2$ assigns the copy type.
• Figure 2: RAPTA pipeline.(a) A pretrained detector highlights saliency-based boxes with class/confidence; we keep the top-$M$ with $S_i>\tau_b$ and discretize each box center to a $3\times3$ grid $\mathcal{G}$ (top-left $\cdots$ bottom-right), yielding position tokens $\mathrm{pos}_i$. (b) From the base prompt $p$, detected class $c_i$ and position $\mathrm{pos}_i$, a small template set $\{T_j\}_{j=1}^{J}$ instantiates region-aware variants $V=\{p\}\cup\{\,T_j(p,c_i,\mathrm{pos}_i)\,\}$. (c) For image $I$ and each $v\in V$, CLIP consistency $S_v=\cos\!(f_I,f_v)$ is converted to weights $w_v=S_v^{\gamma}$ and normalized to $\pi(v)=w_v/\sum_{u\in V}w_u$; we sample a single prompt $\tilde{p}\sim\pi(\cdot)$ to condition training. This preserves prompt–image alignment while injecting grounded diversity.
• Figure 3: ADMCD pipeline.(a) For an image $X\in\{G,R\}$, three complementary embeddings are extracted: a patch-level visual descriptor $\mathbf f^{\mathrm{vis}}(X)$, a global CLIP descriptor $\mathbf f^{\mathrm{clip}}(X)$, and a texture descriptor $\mathbf f^{\mathrm{tex}}(X)$. After linear projection, the streams are fused by a lightweight Transformer and $\ell_2$-normalized to yield $\hat{\mathbf f}_{\mathrm{fus}}(X)$. (b) Copy decision uses fused cosine similarity $S_{\mathrm{fus}}$; we flag Copy if $S_{\mathrm{fus}}>\tau_1$. (c) For flagged pairs we compute stream-wise similarities $S_{\mathrm{vis}}, S_{\mathrm{clip}}, S_{\mathrm{tex}}$ and the similarity-weighted score $\bar{S}$; Retrieve/Exact if $\bar{S}>\tau_2$, otherwise Style. Thresholds $(\tau_1,\tau_2)$ and weights $(\omega_1,\omega_2,\omega_3)$ are fixed from validation sweeps.
• Figure 4: Accuracy vs. threshold on fused similarity $S_{\mathrm{fus}}$. Best accuracy is achieved at $\tau=0.938$.
• Figure 5: Accuracy heatmap over $(w_{\mathrm{vis}},w_{\mathrm{clip}})$ with $w_{\mathrm{tex}}=1-w_{\mathrm{vis}}-w_{\mathrm{clip}}$. Peak at $(0.24,0.38,0.38)$.