VLM-Guided Adaptive Negative Prompting for Creative Generation

TL;DR

The paper tackles the challenge of enabling diffusion-based generation to yield truly novel yet coherent visuals by addressing the tension between mode coverage and mode seeking. It introduces VLM-Guided Adaptive Negative Prompting, a training-free, inference-time loop where a Vision-Language Model analyzes intermediate denoising outputs to accumulate dynamic negative prompts, steering the trajectory away from conventional concepts while preserving category identity. The method integrates into standard diffusion pipelines with minimal runtime overhead and demonstrates consistent gains in novelty and diversity, validated by both human studies and quantitative CLIP/GPT-based metrics. Its ability to generate coherent creative sets and handle complex compositional prompts suggests a practical, scalable route to exploratory creativity in multimodal generation, with potential extensions to video, 3D, and multimodal content.

Abstract

Creative generation is the synthesis of new, surprising, and valuable samples that reflect user intent yet cannot be envisioned in advance. This task aims to extend human imagination, enabling the discovery of visual concepts that exist in the unexplored spaces between familiar domains. While text-to-image diffusion models excel at rendering photorealistic scenes that faithfully match user prompts, they still struggle to generate genuinely novel content. Existing approaches to enhance generative creativity either rely on interpolation of image features, which restricts exploration to predefined categories, or require time-intensive procedures such as embedding optimization or model fine-tuning. We propose VLM-Guided Adaptive Negative-Prompting, a training-free, inference-time method that promotes creative image generation while preserving the validity of the generated object. Our approach utilizes a vision-language model (VLM) that analyzes intermediate outputs of the generation process and adaptively steers it away from conventional visual concepts, encouraging the emergence of novel and surprising outputs. We evaluate creativity through both novelty and validity, using statistical metrics in the CLIP embedding space. Through extensive experiments, we show consistent gains in creative novelty with negligible computational overhead. Moreover, unlike existing methods that primarily generate single objects, our approach extends to complex scenarios, such as generating coherent sets of creative objects and preserving creativity within elaborate compositional prompts. Our method integrates seamlessly into existing diffusion pipelines, offering a practical route to producing creative outputs that venture beyond the constraints of textual descriptions.

Figures (18)

• Figure 1: Images generated with GPT-o3 gpt-o3, GPT-4o openai2024gpt4ocard, SDXL podell2023sdxlimprovinglatentdiffusion, FLUX-dev flux, and SD3.5 esser2024scaling using the prompt "Professional high-quality photo of a new type of pet."
• Figure 2: Overview of our VLM-guided negative prompting method. To generate a creative image (e.g., "new type of pet"), we sample Gaussian noise and perform an augmented denoising process that maintains an adaptive list of negative prompts. At each denoising step, we query a pre-trained Vision-Language Model (VLM) to identify visual concepts present in the intermediate output and update the list accordingly, steering the denoising process away from them. For example, we add the token "cat" to the accumulating list to shift the denoising trajectory away from generating an image resembling a cat as well as the previously detected pets.
• Figure 3: Trade-off between novelty and category coherence in our user study. Higher values are better for both axes. Our method (star) uniquely achieves high scores on both dimensions compared to other creative generation methods.
• Figure 4: Qualitative results of our method across different object categories. In all categories, our method generates creative shapes and appearances while preserving object semantics. For instance, buildings with unique forms and textures that retain windows, doors, and balconies, or bags made of varied materials that remain recognizable as bags.
• Figure 5: Left: Comparison with ConceptLab Richardson_2024 (top row) and our VLM-Guided method using Kandinsky2 razzhigaev2023kandinskyimprovedtexttoimagesynthesis (middle row) and SD3.5 (bottom row). Right: Comparison with C3 han2025enhancing using SDXL podell2023sdxlimprovinglatentdiffusion (top row) and our method using SDXL (middle row) and SD3.5 (bottom row). Our method consistently generates more diverse and imaginative variations while maintaining recognizability within each category.