Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

T2I-FineEval: Fine-Grained Compositional Metric for Text-to-Image Evaluation

Seyed Mohammad Hadi Hosseini, Amir Mohammad Izadi, Ali Abdollahi, Armin Saghafian, Mahdieh Soleymani Baghshah

TL;DR

The paper tackles the challenge of evaluating compositionality in text-to-image generation, where existing metrics like CLIPScore fail to capture attribute binding and spatial relations. It introduces T2I-FineEval, a fine-grained metric that decomposes both the prompt and the generated image: prompts are broken into entity, relational, and general assertions, while images are parsed into entity boxes and relational boxes via object detection. A VQA model scores question-box pairs, with a FILIP-like matching step to align questions with image components, and the final score is the mean of fine-grained and coarse-grained evaluations: $score(\\mathcal{I}, \\\mathcal{P}) = \\frac{\\text{fine-grained score}(\\mathcal{I}, \\\mathcal{P}) + \\text{coarse-grained score}(\\mathcal{I}, \\\mathcal{P})}{2}$. Through experiments on the T2I-CompBench benchmark across six models and multiple metrics, T2I-FineEval demonstrates stronger correlation with human judgments, indicating improved reliability for assessing compositional accuracy in text-to-image generation. The work suggests a practical path toward more human-aligned evaluation and provides code for reproducibility. $

Abstract

Although recent text-to-image generative models have achieved impressive performance, they still often struggle with capturing the compositional complexities of prompts including attribute binding, and spatial relationships between different entities. This misalignment is not revealed by common evaluation metrics such as CLIPScore. Recent works have proposed evaluation metrics that utilize Visual Question Answering (VQA) by decomposing prompts into questions about the generated image for more robust compositional evaluation. Although these methods align better with human evaluations, they still fail to fully cover the compositionality within the image. To address this, we propose a novel metric that breaks down images into components, and texts into fine-grained questions about the generated image for evaluation. Our method outperforms previous state-of-the-art metrics, demonstrating its effectiveness in evaluating text-to-image generative models. Code is available at https://github.com/hadi-hosseini/ T2I-FineEval.

T2I-FineEval: Fine-Grained Compositional Metric for Text-to-Image Evaluation

TL;DR

The paper tackles the challenge of evaluating compositionality in text-to-image generation, where existing metrics like CLIPScore fail to capture attribute binding and spatial relations. It introduces T2I-FineEval, a fine-grained metric that decomposes both the prompt and the generated image: prompts are broken into entity, relational, and general assertions, while images are parsed into entity boxes and relational boxes via object detection. A VQA model scores question-box pairs, with a FILIP-like matching step to align questions with image components, and the final score is the mean of fine-grained and coarse-grained evaluations: . Through experiments on the T2I-CompBench benchmark across six models and multiple metrics, T2I-FineEval demonstrates stronger correlation with human judgments, indicating improved reliability for assessing compositional accuracy in text-to-image generation. The work suggests a practical path toward more human-aligned evaluation and provides code for reproducibility. $

Abstract

Although recent text-to-image generative models have achieved impressive performance, they still often struggle with capturing the compositional complexities of prompts including attribute binding, and spatial relationships between different entities. This misalignment is not revealed by common evaluation metrics such as CLIPScore. Recent works have proposed evaluation metrics that utilize Visual Question Answering (VQA) by decomposing prompts into questions about the generated image for more robust compositional evaluation. Although these methods align better with human evaluations, they still fail to fully cover the compositionality within the image. To address this, we propose a novel metric that breaks down images into components, and texts into fine-grained questions about the generated image for evaluation. Our method outperforms previous state-of-the-art metrics, demonstrating its effectiveness in evaluating text-to-image generative models. Code is available at https://github.com/hadi-hosseini/ T2I-FineEval.

Paper Structure

This paper contains 9 sections, 6 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. Evaluated models and Metrics
  5. Benchmarks
  6. Implementation Details
  7. Human Correlation of the Evaluation Metrics
  8. Result Analysis
  9. Conclusion

Figures (1)

  • Figure 1: Illustration of T2I-FineEval: A Large Language Model generates coarse-grained, entity-specific, and relational questions from a text prompt. An object detector identifies entities in an image created from the same prompt, forming boxes for each entity and pairing them to capture candidate relationships (relational boxes). A Visual Question-Answering (VQA) model is used to ask coarse-grained questions about the entire image, resulting in a coarse-grained score. Entity-specific questions are matched with their corresponding entity boxes, while relational questions are paired with relational boxes, forming a question-box matrix. The highest scoring box for each question is selected, and these scores contribute to an image's fine-grained score. The overall score is then the average of the fine-grained and coarse-grained scores.