Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers

TL;DR

The work investigates MIS in diffusion-transformer–based text-to-image generation, revealing a hierarchical, layer- and step-dependent attention structure in DiT models. It introduces a training-free Hierarchical and Step-Layer-Wise Attention Specialty Tuning (AST) with a Hierarchical and Step-Layer-Wise (HSLW) design to selectively amplify or suppress attention regions using sketch-guided masks. The method demonstrates substantial MIS improvements on upgraded sketch-based benchmarks and customized scenes, with ablations highlighting the distinct roles of T2T, I2I, and I2T attentions and demonstrating robustness across DiT variants (e.g., FLUX and SD v3.5). Overall, AST/HSLW offers a practical, model-agnostic approach to achieve precise instance placement and attribute rendering in complex multi-instance prompts, enabling more controllable MIS without fine-tuning. The framework also provides an analysis-design-experiment paradigm for future MIS enhancements in diffusion transformers.

Abstract

Text-to-image (T2I) generation models often struggle with multi-instance synthesis (MIS), where they must accurately depict multiple distinct instances in a single image based on complex prompts detailing individual features. Traditional MIS control methods for UNet architectures like SD v1.5/SDXL fail to adapt to DiT-based models like FLUX and SD v3.5, which rely on integrated attention between image and text tokens rather than text-image cross-attention. To enhance MIS in DiT, we first analyze the mixed attention mechanism in DiT. Our token-wise and layer-wise analysis of attention maps reveals a hierarchical response structure: instance tokens dominate early layers, background tokens in middle layers, and attribute tokens in later layers. Building on this observation, we propose a training-free approach for enhancing MIS in DiT-based models with hierarchical and step-layer-wise attention specialty tuning (AST). AST amplifies key regions while suppressing irrelevant areas in distinct attention maps across layers and steps, guided by the hierarchical structure. This optimizes multimodal interactions by hierarchically decoupling the complex prompts with instance-based sketches. We evaluate our approach using upgraded sketch-based layouts for the T2I-CompBench and customized complex scenes. Both quantitative and qualitative results confirm our method enhances complex layout generation, ensuring precise instance placement and attribute representation in MIS.

Figures (23)

• Figure 1: Schematic architecture of the DiT-based models (a) with double/single-stream attentions (b). The attention map in (c) reveals four distinct interaction patterns: text-to-text, text-to-image, image-to-text, and image-to-image, enabling deep consistent fusion between text and tokens.
• Figure 2: Attention map averages for the prompt "Red cube in a forest". (a) T2T maps show strong intra-segment interactions within valid tokens of "Red cube" (first 3 tokens) and "in a forest" (last 4 tokens). (b) I2I maps (the maximum value of each point relative to all other points) reveal growing interaction between "Red cube" and "a forest" segments over steps. (c/d/e) I2T attention maps on individual tokens highlight that instance tokens dominate early layers, background tokens in the middle, and color tokens later, with most information integrated in the first half of steps. T2I maps showing lower scores with weak impact and more examples are present in the Appendix.\ref{['app-section:8']}.
• Figure 3: Step-Layer-Wise Token Exchange involved injecting specific tokens between two prompts formatted as "[color] [Instance] in [Background]" (a). First 6-step exchanging results show that specific token concept can be replaced in certain layers (e.g., L25-27) with other concepts preserving, aligning with the hierarchical attention responses in Figure \ref{['fig:self']}. Step-wise visualization is detailed in Appendix.\ref{['app-section:9']}.
• Figure 4: Overview of the hierarchical tuning strategy for DiT models. (a) Attention specialty tuning enhances meaningful regions while suppressing non-meaningful ones via a unified scalable module with distinct tuning masks. (b) Tuning masks for T2T and I2I masks are directly built from prompt segments and sketches. (c) Hierarchical and step-layer-wise module refines alignment by adjusting each component’s impact across specific layers and steps, assigning layers for attributes, instances, and background tokens within first 16 steps.
• Figure 5: Visualizations on customized complex cases (More examples are in Appendix.\ref{['app-section:6']}). Bounding box layouts are for MIGC zhou2024migc Instance wang2024instancediffusion, R&B xiao2023r, and GroundDiT lee2024groundit, and the layout of RPG yang2024mastering is generated by the GPT-4o openai2024gpt4technicalreport, while sketch layouts are for ours. Our approach extends DiT models (SD v3.5 and FLUX) with better multi-instance alignment with prompts and sketch-based layouts. The underlined prompts within the prompts are the sub-prompts for instances, with colors matching those in the sketch.