CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback

TL;DR

CompAlign addresses a key gap in text-to-image evaluation by introducing a demanding benchmark of 900 prompts that stress 3D-spatial reasoning, numeracy, and attribute binding. The authors propose CompQuest, a transparent, three-step evaluation framework that decomposes prompts into atomic questions, uses an MLLM for binary feedback, and computes Aggregated Compositional Accuracy to quantify alignment. They show significant performance gaps between open-source and closed-source T2I models and demonstrate that a CompQuest-based alignment pipeline can meaningfully improve compositional accuracy for diffusion models. The work delivers a scalable methodology for both evaluating and improving compositional T2I generation, with practical implications for design, entertainment, and other real-world applications.

Abstract

State-of-the-art T2I models are capable of generating high-resolution images given textual prompts. However, they still struggle with accurately depicting compositional scenes that specify multiple objects, attributes, and spatial relations. We present CompAlign, a challenging benchmark with an emphasis on assessing the depiction of 3D-spatial relationships, for evaluating and improving models on compositional image generation. CompAlign consists of 900 complex multi-subject image generation prompts that combine numerical and 3D-spatial relationships with varied attribute bindings. Our benchmark is remarkably challenging, incorporating generation tasks with 3+ generation subjects with complex 3D-spatial relationships. Additionally, we propose CompQuest, an interpretable and accurate evaluation framework that decomposes complex prompts into atomic sub-questions, then utilizes a MLLM to provide fine-grained binary feedback on the correctness of each aspect of generation elements in model-generated images. This enables precise quantification of alignment between generated images and compositional prompts. Furthermore, we propose an alignment framework that uses CompQuest's feedback as preference signals to improve diffusion models' compositional image generation abilities. Using adjustable per-image preferences, our method is easily scalable and flexible for different tasks. Evaluation of 9 T2I models reveals that: (1) models remarkable struggle more with compositional tasks with more complex 3D-spatial configurations, and (2) a noticeable performance gap exists between open-source accessible models and closed-source commercial models. Further empirical study on using CompAlign for model alignment yield promising results: post-alignment diffusion models achieve remarkable improvements in compositional accuracy, especially on complex generation tasks, outperforming previous approaches.

Figures (7)

• Figure 1: Example showing that while T2I models are capable of depicting simple compositional settings in a previous benchmark, they fall short in correctly generating images that accurately conform to complex compositional instructions in our challenging CompAlign benchmark.
• Figure 2: The CompQuest Evaluation Framework. Each compositional prompt is first divided into atomic questions, then answered by a MLLM feedback model, finally aggregated to be an accuracy score.
• Figure 3: Visualization of evaluation results, stratified by generation types. We observe that the older generation of UNet-based diffusion models fall short in achieving good compositional performance for tasks such as depicting the color attribute bindings for objects.
• Figure 4: Qualitative examples of how CompAlign-ed diffusion model demonstrate better performance on compositional T2I tasks.
• Figure 5: Visualization of experiment results on SD2. Our alignment method effectively improves the compositional ability of diffusion models on most generation aspects, whereas 10847875's checkpoint frequently underperforms even the base model.