ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation

Abstract

Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.

Figures (14)

• Figure 1: Qualitative comparison between R2F and ADAPT (ours) across rare concepts. Our proposed method significantly enhances R2F in a zero-shot way, demonstrating superior capability in text-image alignment. All the pairs are generated with the same seed (42).
• Figure 2: Overview of our ADAPT framework. (a) Our framework introduces three complementary zero-shot control modules in the Stable Diffusion 3 (SD3) architecture. Adaptive Prompt Scheduling (APS) determines optimal stop points $S^i$ based on each token's spatial attention map score $\mathcal{S}_{\text{Attn}}$ at each step; Pooled Embedding Manipulation (PEM) operates on the CLIP's pooled text embedding prior to modulation in the SD3 pipeline; Latent Space Manipulation (LSM) is applied at the feature level directly within the transformer block after attention computation. (b) The overview of attention scoring. We compute Top-k attention response scores $s^{(k)}$ to determine when each token's semantic content has been sufficiently established. (c) Visualization of adaptive prompt scheduling between $y_\text{prog}$ and target prompt $y_\text{tar}$ based on stop points $S^i$.
• Figure 3: Qualitative comparison between previous methods and our method across rare concept prompts from RareBench. Our approach significantly improves upon R2F in a zero-shot setting, demonstrating superior alignment between image and text for a wide range of challenging attributes and compositions. All samples are generated with the same random seed (42) for fair comparison.
• Figure 4: Qualitative comparison on the hyperparameters introduced within the ADAPT framework.
• Figure 5: Attention response score $\mathcal{S}_{\text{Attn}}$ of target prompt "A horned pelican".