UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation

TL;DR

UniGenBench++ addresses the limitations of existing T2I benchmarks by offering a unified, fine-grained semantic evaluation framework with 600 prompts organized hierarchically across 5 real-world themes and 20 subthemes, evaluated on 10 primary and 27 sub-dimensions. Prompts are provided in English and Chinese in short and long forms, enabling robust testing of language and prompt-length sensitivity via a streamlined MLLM-based pipeline using Gemini-2.5-Pro, plus an offline evaluator trained on ~375K samples. The study benchmarks a wide range of open- and closed-source models, revealing that models excel in style and world knowledge but struggle with complex logical reasoning and grammar, with open-source models showing greater performance variability. The offline evaluation model enables cost-efficient, offline assessments, making the benchmark practical for the community and capable of driving deeper, real-world semantic evaluation of T2I systems.

Abstract

Recent progress in text-to-image (T2I) generation underscores the importance of reliable benchmarks in evaluating how accurately generated images reflect the semantics of their textual prompt. However, (1) existing benchmarks lack the diversity of prompt scenarios and multilingual support, both essential for real-world applicability; (2) they offer only coarse evaluations across primary dimensions, covering a narrow range of sub-dimensions, and fall short in fine-grained sub-dimension assessment. To address these limitations, we introduce UniGenBench++, a unified semantic assessment benchmark for T2I generation. Specifically, it comprises 600 prompts organized hierarchically to ensure both coverage and efficiency: (1) spans across diverse real-world scenarios, i.e., 5 main prompt themes and 20 subthemes; (2) comprehensively probes T2I models' semantic consistency over 10 primary and 27 sub evaluation criteria, with each prompt assessing multiple testpoints. To rigorously assess model robustness to variations in language and prompt length, we provide both English and Chinese versions of each prompt in short and long forms. Leveraging the general world knowledge and fine-grained image understanding capabilities of a closed-source Multi-modal Large Language Model (MLLM), i.e., Gemini-2.5-Pro, an effective pipeline is developed for reliable benchmark construction and streamlined model assessment. Moreover, to further facilitate community use, we train a robust evaluation model that enables offline assessment of T2I model outputs. Through comprehensive benchmarking of both open- and closed-sourced T2I models, we systematically reveal their strengths and weaknesses across various aspects.

Figures (4)

• Figure 2: Benchmark Statistics. (a) Word clouds for English and Chinese prompts in both short and long forms; (b) overall prompt length distribution; and (c) distribution of testpoint counts per prompt for short versus long versions.
• Figure 3: Qualitative Results of Evaluation Dimensions. We present qualitative examples of T2I models evaluated across our specified dimensions.
• Figure 4: Pipeline of Benchmark Construction and Offline Evaluation Model Training. (a) Benchmark construction pipeline; (b) Offline evaluation model training; (c) Offline evaluation cases.
• Figure 5: Evaluation Accuracy Comparison. Our dedicated evaluation model demonstrates a significant improvement in evaluation accuracy across all test points compared to the commonly used offline evaluation VLM, i.e., Qwen2.5-VL-72b.