M$^{3}$T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark

TL;DR

This work introduces M$^{3}$T2IBench, a large-scale, multi-category, multi-instance, multi-relation text-to-image benchmark, and AlignScore, an object-detection-based metric that correlates strongly with human judgments for image-text alignment. It demonstrates that existing open- and closed-source diffusion models struggle to follow complex prompts, revealing distinct challenges in multi-instance, multi-category, and multi-relational settings. The paper also presents Revise-Then-Enforce, a training-free post-editing method that constructs paired prompts to refine generation toward text guidance, achieving consistent improvements across models. Collectively, these contributions provide a robust framework for evaluating and enhancing image-text alignment in diffusion-based T2I systems, with data and code to be released after acceptance.

Abstract

Text-to-image models are known to struggle with generating images that perfectly align with textual prompts. Several previous studies have focused on evaluating image-text alignment in text-to-image generation. However, these evaluations either address overly simple scenarios, especially overlooking the difficulty of prompts with multiple different instances belonging to the same category, or they introduce metrics that do not correlate well with human evaluation. In this study, we introduce M$^3$T2IBench, a large-scale, multi-category, multi-instance, multi-relation along with an object-detection-based evaluation metric, $AlignScore$, which aligns closely with human evaluation. Our findings reveal that current open-source text-to-image models perform poorly on this challenging benchmark. Additionally, we propose the Revise-Then-Enforce approach to enhance image-text alignment. This training-free post-editing method demonstrates improvements in image-text alignment across a broad range of diffusion models. \footnote{Our code and data has been released in supplementary material and will be made publicly available after the paper is accepted.}

Figures (12)

• Figure 1: A failure case generated by Stable-Diffusion-3. The right is the prompt and the left is the generated image. We label the parts incorrectly generated in red and questionable parts in yellow.
• Figure 2: Example of data in our benchmark.
• Figure 3: Overview of our benchmark construction and metric calculation.
• Figure 4: Statistics of our benchmark.
• Figure 5: Average $Acc$ and $Bias$ under different number of categories or number of relations. Total number of instances is fixed as $4$.