ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation

TL;DR

ProCreate introduces propulsive energy diffusion, an energy-based guidance mechanism that pushes generated embeddings away from a reference set during diffusion sampling to boost creativity and mitigate data replication. It combines a Multi-Step Look Ahead prediction of the clean image with a dynamically growing reference set and uses DreamSim embeddings to compute cosine-based similarities, controlled by a strength parameter $\gamma$. On FSCG-8, a new eight-category, few-shot dataset, ProCreate achieves superior diversity and competitive fidelity compared with standard DDIM and CADS, and it consistently reduces the likelihood of replicating training data in large-scale replication experiments (e.g., LAION12M) while preserving quality metrics like FID and KID. The work demonstrates that simple, inference-time guidance can substantially enhance creative sampling and data-privacy properties, with broader implications for designer workflows and responsible deployment of generative AI.

Abstract

In this paper, we propose ProCreate, a simple and easy-to-implement method to improve sample diversity and creativity of diffusion-based image generative models and to prevent training data reproduction. ProCreate operates on a set of reference images and actively propels the generated image embedding away from the reference embeddings during the generation process. We propose FSCG-8 (Few-Shot Creative Generation 8), a few-shot creative generation dataset on eight different categories -- encompassing different concepts, styles, and settings -- in which ProCreate achieves the highest sample diversity and fidelity. Furthermore, we show that ProCreate is effective at preventing replicating training data in a large-scale evaluation using training text prompts. Code and FSCG-8 are available at https://github.com/Agentic-Learning-AI-Lab/procreate-diffusion-public. The project page is available at https://procreate-diffusion.github.io.

Figures (11)

• Figure 1: After fine-tuning a diffusion model on each category of our few-shot dataset FSCG-8, ProCreate can significantly improve the diversity and creativity of generations while retaining high image quality and prompt fidelity.
• Figure 2: Overview of our approach. At each denoising step, ProCreate applies gradient guidance that maximizes the distances between the generated clean image and the reference images in the embedding space of a similarity embedding network. In the embedding space, the noisy image is propelled away from its closest reference image.
• Figure 3: Samples of the FSCG-8 dataset. We provide $10$ samples from each of the $8$ categories. For each category, an example caption for its top left image is provided.
• Figure 4: Qualitative comparison between DDIM, CADS, and ProCreate for few-shot creative generation on FSCG-8 with standard fine-tuning. For each sampling method, we show two prompts and four generated samples for each prompt. In addition, we match each sample from ProCreate with its closest training image based on the SSCD score sscd between the matched pair.
• Figure 5: Compare DDIM, CADS, and ProCreate's performance under different numbers of fine-tuning iterations.