Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning

TL;DR

This work tackles the complete-text comprehension defect in diffusion transformer-based text-to-image generation, where long, syntactically complex captions hinder faithful grounding. It introduces DiT-ST, a split-text conditioning framework that uses an LLM to parse captions into a hierarchical graph and then converts it into a split-text caption, which is fed into a refined DiT backbone with incremental cross-attention-based injection of object, relation, and attribute primitives at carefully chosen diffusion timesteps. The approach leverages adaptive injection timing, determined by cross-attention convergence and SNR inflection points, to improve stage-specific semantic representation and reduce premature information exposure. Empirical results on GenEval and COCO-5K show that DiT-ST achieves higher semantic fidelity (GenEval ~0.69; CLIPScore ~34.09) and competitive image fidelity (FID ~22.11) compared with strong baselines, demonstrating robustness to caption length and applicability across architectures with modest parameter overhead.

Abstract

Current text-to-image diffusion generation typically employs complete-text conditioning. Due to the intricate syntax, diffusion transformers (DiTs) inherently suffer from a comprehension defect of complete-text captions. One-fly complete-text input either overlooks critical semantic details or causes semantic confusion by simultaneously modeling diverse semantic primitive types. To mitigate this defect of DiTs, we propose a novel split-text conditioning framework named DiT-ST. This framework converts a complete-text caption into a split-text caption, a collection of simplified sentences, to explicitly express various semantic primitives and their interconnections. The split-text caption is then injected into different denoising stages of DiT-ST in a hierarchical and incremental manner. Specifically, DiT-ST leverages Large Language Models to parse captions, extracting diverse primitives and hierarchically sorting out and constructing these primitives into a split-text input. Moreover, we partition the diffusion denoising process according to its differential sensitivities to diverse semantic primitive types and determine the appropriate timesteps to incrementally inject tokens of diverse semantic primitive types into input tokens via cross-attention. In this way, DiT-ST enhances the representation learning of specific semantic primitive types across different stages. Extensive experiments validate the effectiveness of our proposed DiT-ST in mitigating the complete-text comprehension defect.

Figures (11)

• Figure 1: (a) Images generated by MM-DiT 8B-E using different forms of the same caption. Our split-text caption enables the model to notice details, such as unripe and partly green, pink ribbon bow, and display them in the generated image. (b) Comparison of text-to-image generation results between Stable Diffusion v3 and ours. Obviously, ours has better semantic details and mitigates the attribute misbinding such as green car.
• Figure 2: (a) Attention maps tang2022daam for various semantic primitives. Caption: A teddy bear wearing a red ribbon around its neck. Attentions exhibit significant overlap between the object primitive 'ribbon' and relation primitive 'wears', resulting in semantic entanglement. (b) Superimposed attention maps of object primitive type and relation primitive type at denoising timesteps 25 and 75, respectively. Notably, the model focuses more on object primitives during the earlier stage and shifts more attention to relation primitives in the later stage.
• Figure 3: The overall framework of DiT-ST. Three colors represent object, relation and attribute, respectively.
• Figure 4: DiT text encoding refinement.
• Figure 5: Inflation point of SNR.