Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models

TL;DR

The paper systematically evaluates compositional generation in nine state-of-the-art text-to-image backbones (seven diffusion-based, two autoregressive) using the T2I-CompBench benchmark. It finds that vanilla autoregressive models like LlamaGen underperform diffusion models of comparable size, while open-source diffusion models such as FLUX reach parity with the closed-source DALL-E3. Pixart-$oldsymboleta$alpha also demonstrates competitive performance, sometimes surpassing SD-XL on several tasks. The results highlight the need for inductive biases beyond next-token prediction and suggest tokenizer design and diffusion-transformer hybrids as promising directions to improve compositional fidelity in T2I systems.

Abstract

Text-to-image (T2I) generative models, such as Stable Diffusion and DALL-E, have shown remarkable proficiency in producing high-quality, realistic, and natural images from textual descriptions. However, these models sometimes fail to accurately capture all the details specified in the input prompts, particularly concerning entities, attributes, and spatial relationships. This issue becomes more pronounced when the prompt contains novel or complex compositions, leading to what are known as compositional generation failure modes. Recently, a new open-source diffusion-based T2I model, FLUX, has been introduced, demonstrating strong performance in high-quality image generation. Additionally, autoregressive T2I models like LlamaGen have claimed competitive visual quality performance compared to diffusion-based models. In this study, we evaluate the compositional generation capabilities of these newly introduced models against established models using the T2I-CompBench benchmark. Our findings reveal that LlamaGen, as a vanilla autoregressive model, is not yet on par with state-of-the-art diffusion models for compositional generation tasks under the same criteria, such as model size and inference time. On the other hand, the open-source diffusion-based model FLUX exhibits compositional generation capabilities comparable to the state-of-the-art closed-source model DALL-E3.