Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement
Chenghao Li, Dake Chen, Yuke Zhang, Peter A. Beerel
TL;DR
Diffusion models risk memorizing training data, leading to data replication and privacy concerns. The authors introduce a generality score for captions, employ an LLM to generalize training captions, and propose a dual fusion enhancement that blends latent representations and caption conditioning during training to mitigate replication. Empirical results show up to 43.5% replication reduction versus the baseline while maintaining generation diversity and quality, with 5-word generalized captions yielding notable gains (32.18% reduction) and the dual fusion providing additional improvements (~16.7%). This work offers practical privacy improvements for diffusion-based generation and lays groundwork for guiding caption generalization with a quantified generality metric.
Abstract
While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to `replicate' training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5% compared to the original diffusion model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.