Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

TL;DR

This work systematically evaluates deep fusion between a frozen large language model and a trainable diffusion transformer for text-to-image synthesis. By conducting controlled comparisons with shallow fusion baselines and probing design choices—including timestep conditioning, positional encodings, and base LLM capabilities—it demonstrates that deep fusion improves text–image alignment and competitiveness on standard benchmarks, while allowing efficient inference. The study also provides a scalable, reproducible training recipe and shows that architecture can be decoupled across modalities, enabling independent scaling. Overall, the paper offers practical guidelines and data-driven insights to advance multi-modal generation by tightly coupling autoregressive language modeling with diffusion-based image synthesis.

Abstract

This paper does not describe a new method; instead, it provides a thorough exploration of an important yet understudied design space related to recent advances in text-to-image synthesis -- specifically, the deep fusion of large language models (LLMs) and diffusion transformers (DiTs) for multi-modal generation. Previous studies mainly focused on overall system performance rather than detailed comparisons with alternative methods, and key design details and training recipes were often left undisclosed. These gaps create uncertainty about the real potential of this approach. To fill these gaps, we conduct an empirical study on text-to-image generation, performing controlled comparisons with established baselines, analyzing important design choices, and providing a clear, reproducible recipe for training at scale. We hope this work offers meaningful data points and practical guidelines for future research in multi-modal generation.

Figures (7)

• Figure 1: Illustration of the deep fusion approach and baselines. We conduct controlled comparisons with baseline methods that incorporate text representations from a single text encoder layer into each DiT layer using late fusion within the attention mechanism, a strategy we term as the "shallow fusion" approach.
• Figure 2: Illustration of the attention mask. Each dotted square indicates whether the row can attend to the column.
• Figure 3: Illustration of cross-modal attention in the shallow fusion baselines. The key and query states of the condition are directly projected from text representations.
• Figure 4: Illustration of timestep conditioning strategies. Removing timesetp conditioning leads to the fewest parameters and the best overall performance.
• Figure 5: Illustration of RoPE. The indices denote position IDs.