Enhancing Creative Generation on Stable Diffusion-based Models

TL;DR

The paper addresses the limited creativity of diffusion models by introducing C3, a training-free method that selectively amplifies low-frequency features in early U-Net blocks during denoising to promote novel outputs. By operating in the Fourier domain with block-specific amplification factors chosen to maximize usability and novelty, C3 achieves higher novelty and diversity with modest impacts on usability, without additional training costs. The approach is demonstrated across SDXL and its distilled variants, integrates with ControlNet, and extends to non–SD models, underscoring its practical potential for designers and artists. Overall, C3 offers a lightweight, broadly applicable catalyst for creative image generation in diffusion models, while acknowledging limitations related to model-specific creativity and architecture compatibility.

Abstract

Recent text-to-image generative models, particularly Stable Diffusion and its distilled variants, have achieved impressive fidelity and strong text-image alignment. However, their creative capability remains constrained, as including `creative' in prompts seldom yields the desired results. This paper introduces C3 (Creative Concept Catalyst), a training-free approach designed to enhance creativity in Stable Diffusion-based models. C3 selectively amplifies features during the denoising process to foster more creative outputs. We offer practical guidelines for choosing amplification factors based on two main aspects of creativity. C3 is the first study to enhance creativity in diffusion models without extensive computational costs. We demonstrate its effectiveness across various Stable Diffusion-based models.

Figures (29)

• Figure 1: Original vs C3 (Ours). Compared to the original diffusion models, Our C3 consistently generates more creative images with no added computational cost. Code is available at https://github.com/daheekwon/C3.
• Figure 2: An overview of the proposed C3 algorithm. We selectively amplify the low-frequency feature of the shallow blocks to enhance creative generations of the pretrained diffusion models.
• Figure 3: Block-wise feature amplification results. All frequency bands are amplified.
• Figure 4: (Top) Uniform amplification across all frequency-band features in the first down block. (Bottom) Amplification of low-frequency features. Enhancing only low-frequency features helps eliminate noise and mosaic patterns.
• Figure 5: The image generated with the automatically selected amplification factors for each block and the combined amplification of all blocks.