Creative Image Generation with Diffusion Model

TL;DR

This work tackles the problem of reproducibility and reliability of creativity in text-to-image diffusion by reframing creativity as exploration of low-probability regions in the embedding space. It introduces a probabilistic framework built on a diffusion prior (sampling embeddings), a creative optimization that pushes embeddings toward distribution tails, and pullback mechanisms (anchor loss and multimodal semantic checks) to preserve semantic fidelity. Directionality is achieved via negative cluster modeling to avoid unappealing directions, enabling efficient and robust exploration of novel visuals. Empirical results demonstrate faster convergence to creative outputs than baselines like ConceptLab, with strong human judgments supporting higher perceived creativity and maintained quality. Overall, the approach provides a principled path toward more expressive and imaginative generative AI without manual subclass exclusion.

Abstract

Creative image generation has emerged as a compelling area of research, driven by the need to produce novel and high-quality images that expand the boundaries of imagination. In this work, we propose a novel framework for creative generation using diffusion models, where creativity is associated with the inverse probability of an image's existence in the CLIP embedding space. Unlike prior approaches that rely on a manual blending of concepts or exclusion of subcategories, our method calculates the probability distribution of generated images and drives it towards low-probability regions to produce rare, imaginative, and visually captivating outputs. We also introduce pullback mechanisms, achieving high creativity without sacrificing visual fidelity. Extensive experiments on text-to-image diffusion models demonstrate the effectiveness and efficiency of our creative generation framework, showcasing its ability to produce unique, novel, and thought-provoking images. This work provides a new perspective on creativity in generative models, offering a principled method to foster innovation in visual content synthesis.

Figures (26)

• Figure 1: Creative Generation from our method for building and vehicle, taking only 2 minutes.
• Figure 2: Overall Model Structure. We first sample a distribution (green cluster) of the generated image embeddings $e$ from the diffusion prior $\epsilon_\theta$ (top). Then, during creative optimization (bottom), we optimize learned token and LoRA layers with creative loss to push the generated embeddings (red dot) toward low-probability regions (orange arrow), constrained by an anchor loss and validated by a multimodal LLM. The diffusion decoder finally renders the resulting images.
• Figure 3: Visualization of the distribution of $e$ shift over training iterations. Green cluster is the default distribution from prior sampling stage (Sec.\ref{['PPS']}). Red cluster is the current distribution. The generated images progressively move toward low-probability regions, resulting in more creative outputs over time. (Zoom In)
• Figure 4: User-rated creativity scores over training iterations for the subject "alien." The observed pattern follows the arousal potential curve, demonstrating how our method (pullback disabled) initially enhances creativity effectively before overshooting. We show visual samples for 3 seeds, omitting default ones (iter 0).
• Figure 5: Generated creative images for building and chair.