Segmentation-Free Guidance for Text-to-Image Diffusion Models

TL;DR

This work introduces segmentation-free guidance, a training-free paradigm that enhances text-to-image diffusion quality by dynamically tailoring the negative prompt for each image patch using the model's own cross-attention signals. By switching from classifier-free guidance to a patch-aware segmentation-free score after $t_s$ iterations and incorporating a segmentation-free scale $a$ along with a guidance strength $\bar{w}$, the approach achieves more faithful local rendering of prompt concepts without retraining. Objective metrics show mixed results (FID/CLIP/IS) while human-aligned metrics (PickScore) and subjective evaluations favor segmentation-free guidance, indicating improved perceptual quality in many cases. The method offers a practical, low-overhead avenue to refine diffusion outputs and includes a principled evaluation strategy on MS-COCO-30K prompts to balance diversity and assessment efficiency.

Abstract

We introduce segmentation-free guidance, a novel method designed for text-to-image diffusion models like Stable Diffusion. Our method does not require retraining of the diffusion model. At no additional compute cost, it uses the diffusion model itself as an implied segmentation network, hence named segmentation-free guidance, to dynamically adjust the negative prompt for each patch of the generated image, based on the patch's relevance to concepts in the prompt. We evaluate segmentation-free guidance both objectively, using FID, CLIP, IS, and PickScore, and subjectively, through human evaluators. For the subjective evaluation, we also propose a methodology for subsampling the prompts in a dataset like MS COCO-30K to keep the number of human evaluations manageable while ensuring that the selected subset is both representative in terms of content and fair in terms of model performance. The results demonstrate the superiority of our segmentation-free guidance to the widely used classifier-free method. Human evaluators preferred segmentation-free guidance over classifier-free 60% to 19%, with 18% of occasions showing a strong preference. Additionally, PickScore win-rate, a recently proposed metric mimicking human preference, also indicates a preference for our method over classifier-free.

Figures (11)

• Figure 1: All images are generated using classifier-free guidance from the same seed and positive prompt, "A dog on a couch in an office," but with different negative prompts (NP). As can be seen, the regions corresponding to the omitted concepts improve in detail.
• Figure 2: Effect of segmentation-free guidance. Prompt: "a cute Maltese white dog next to a cat". The long hair characteristic of the dog spills over to the cat under classifier-free guidance (\ref{['fig:effect_segm_free-a']}). Segmentation-free guidance improves quality by adjusting conditioning for the dog and the cat individually (\ref{['fig:effect_segm_free-b']}).
• Figure 3: Effect of segmentation-free scale ($a$ in \ref{['alg:segm-free']}). Prompt: "portrait of a dog and a kid". $a = 0$ is ineffective due to conditioning leakage (\ref{['fig:effect_a-b']}), while large values $a > 20$ cause artifacts (\ref{['fig:effect_a-d']}, \ref{['fig:effect_a-e']}). Subjective evaluations show that $a = 10$ gives the best results (\ref{['fig:effect_a-c']}).
• Figure 4: Compositional effect of $t_s$. Prompt: "a girl hugging a Corgi on a pedestal". Switching too early from classifier-free to segmentation-free guidance improves local detail, but hurts the overall composition, i.e., too large a dog in (\ref{['fig:effect_ts_comp-c']}).
• Figure 5: Effect of $t_s$ in skipping aspects of prompt. Prompt: "architectural drawing of a new town square for Cambridge England, big traditional museum with columns, fountain in middle, classical design, traditional design, trees". Using a smaller $t_s$ in (\ref{['fig:effect_ts_ignor-c']}) improves the overall image quality with respect to (\ref{['fig:effect_ts_ignor-b']}), but at the expense of ignoring the "architectural drawing" aspect of the prompt.