Text-Anchored Score Composition: Tackling Condition Misalignment in Text-to-Image Diffusion Models

TL;DR

This work tackles condition misalignment in text-to-image diffusion by introducing Text-Anchored Score Composition (TASC), a training-free framework that decomposes extra controls into pairwise alignments and combines them with the unified text condition. A core contribution is an equation that expresses the diffusion-step score as a sum of a unified text-driven term and multiple pairwise terms, augmented by an attention realignment that harmonizes unified and individual scores during sampling: $\epsilon(z_t, \mathcal{P}, \mathbb{I}) = \epsilon_\theta(z_t, \varnothing, \varnothing) + w_0(\epsilon_\theta(z_t, \mathcal{P}, \varnothing) - \epsilon_\theta(z_t, \varnothing, \varnothing)) + \sum_{k=1}^K w_k(\epsilon_{\theta,\phi_k}(z_t, \mathcal{P}_{S(k)}, \mathcal{I}_k) - \epsilon_\theta(z_t, \varnothing, \varnothing))$. Attention maps $M^0$ and $M^k$ are then realigned via $M^0_{S(k)} \leftarrow M^k$ to produce a modified unified score for sampling. Extensive experiments across ControlNet/GLIGEN setups show improved controllability, image quality (lower FID, higher CLIP/BLIP alignment), and favorable user feedback, with ablations confirming the critical role of the realignment step. The approach is compatible with existing adapters and offers a flexible, training-free path to reliable multi-condition image synthesis in partially aligned scenarios.

Abstract

Text-to-image diffusion models have advanced towards more controllable generation via supporting various additional conditions (e.g.,depth map, bounding box) beyond text. However, these models are learned based on the premise of perfect alignment between the text and extra conditions. If this alignment is not satisfied, the final output could be either dominated by one condition, or ambiguity may arise, failing to meet user expectations. To address this issue, we present a training free approach called Text-Anchored Score Composition (TASC) to further improve the controllability of existing models when provided with partially aligned conditions. The TASC firstly separates conditions based on pair relationships, computing the result individually for each pair. This ensures that each pair no longer has conflicting conditions. Then we propose an attention realignment operation to realign these independently calculated results via a cross-attention mechanism to avoid new conflicts when combining them back. Both qualitative and quantitative results demonstrate the effectiveness of our approach in handling unaligned conditions, which performs favorably against recent methods and more importantly adds flexibility to the controllable image generation process. Our code will be available at: https://github.com/EnVision-Research/Decompose-and-Realign.

Figures (9)

• Figure 1: Illustration of our proposed Text-Anchored Score Composition showcasing the ability to handle the misalignment between conditions for controllable generation tasks. For example, with several control conditions (e.g., depth, pose, bounding box) specifying the layout structure and the text condition indicating extra guidance (e.g., new object, new correspondence of spatial relationship), our method is able to generate high-quality plausible outputs that satisfy all given conditions without training.
• Figure 2: Challenges in Multi-Condition Image Synthesis. Left: the "dominance" effect exists across various methods, where one object (e.g., car) overshadows the generation, omitting the other (e.g., dog). Right: The "ambiguity" issue, where it's unclear which object from the text matches the intended depth condition.
• Figure 3: Image Synthesis using Text-Anchored Score Composition. Given a depth map of a street and a textual description specifying "a Superman in the street", our method effectively generates the corresponding image.
• Figure 4: Illustration of the "Attention Realignment" operation. The operation addresses mismatches between the unified score and individual scores during generation, preventing the final object from appearing multiple times, as individual score for the same token "car" will not contribute to the attention value in the red box.
• Figure 5: Applications. Demonstrating scalability, our method adeptly manages complex scenes, seamlessly combining controller mechanisms like both ControlNet zhang2023adding and GLIGEN li2023gligen simultaneously.