TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency

TL;DR

This work tackles the difficulty of aligning text-to-image models with long, detailed prompts by introducing LPG-Bench, a benchmark of 200 long prompts (average >$250$ words) evaluated across $13$ models with $2{,}600$ images and human rankings. It then proposes TIT, a zero-shot, decoupled evaluation framework that separates visual description from textual semantic alignment, with two instantiations: TIT-Score (embedding-based) and TIT-Score-LLM (LLM-based). Empirical results show existing metrics poorly track human preferences for long prompts, while TIT methods yield superior alignment, with TIT-Score-LLM achieving up to approximately $66.5 ext{%}$ pairwise accuracy and a high correlation to human rankings. The work provides a practical, generalizable tool for benchmarking long-prompt T2I systems and advancing models’ deep textual understanding, with resources released publicly.

Abstract

With the rapid advancement of large multimodal models (LMMs), recent text-to-image (T2I) models can generate high-quality images and demonstrate great alignment to short prompts. However, they still struggle to effectively understand and follow long and detailed prompts, displaying inconsistent generation. To address this challenge, we introduce LPG-Bench, a comprehensive benchmark for evaluating long-prompt-based text-to-image generation. LPG-Bench features 200 meticulously crafted prompts with an average length of over 250 words, approaching the input capacity of several leading commercial models. Using these prompts, we generate 2,600 images from 13 state-of-the-art models and further perform comprehensive human-ranked annotations. Based on LPG-Bench, we observe that state-of-the-art T2I alignment evaluation metrics exhibit poor consistency with human preferences on long-prompt-based image generation. To address the gap, we introduce a novel zero-shot metric based on text-to-image-to-text consistency, termed TIT, for evaluating long-prompt-generated images. The core concept of TIT is to quantify T2I alignment by directly comparing the consistency between the raw prompt and the LMM-produced description on the generated image, which includes an efficient score-based instantiation TIT-Score and a large-language-model (LLM) based instantiation TIT-Score-LLM. Extensive experiments demonstrate that our framework achieves superior alignment with human judgment compared to CLIP-score, LMM-score, etc., with TIT-Score-LLM attaining a 7.31% absolute improvement in pairwise accuracy over the strongest baseline. LPG-Bench and TIT methods together offer a deeper perspective to benchmark and foster the development of T2I models. All resources will be made publicly available.

Figures (18)

• Figure 1: The construction workflow of LPG-Bench. (a) Long Prompt Generation. 200 core themes are expanded by Gemini 2.5 Pro gemini2.5pro into detailed prompts averaging over 250 words. (b) Image Generation. A suite of text-to-image models produces 2,600 images from these prompts. (c) Human Annotation. The outputs are evaluated through pairwise comparisons, a process that yields 12,832 valid (non-tie) paired results for model ranking.
• Figure 2: Model performance ranking based on human preferences. The left chart shows Average Rank (overall consistency, lower is better), while the right shows the count of #1 Ranks (peak performance, higher is better). The results reveal a clear performance hierarchy among the 13 models.
• Figure 3: A comparison of Text-to-Image evaluation methods. The top row illustrates the diagrams for four baseline methods. The bottom row shows the architecture for our proposed TIT-Score and TIT-Score-LLM. TIT-Score translates an image to a text description via a VLM and then uses a text embedding model to calculate cosine similarity with the prompt; TIT-Score-LLM, however, employs a Large Language Model (LLM) for this final similarity assessment.
• Figure 4: The qualitative case study for two prompts. For both the first and second groups, TIT-Score was the only metric to correctly predict the ranking of all three images. Moreover, its numerical scores provide a clearer indication of the qualitative gap between each image compared to other models.
• Figure 5: Wordcloud of LPG-Bench.