CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checks

TL;DR

CROC introduces an automated meta-evaluation framework for T2I metrics that uses contrastive robustness checks to thoroughly probe metric behavior across a taxonomy of image properties. It delivers a large synthetic dataset (CROC$^{syn}$) and a human-supervised corpus (CROC$^{hum}$), enabling fine-grained comparisons and the training of a new open-source metric, CROCScore, which achieves state-of-the-art performance on several benchmarks. The framework reveals robustness gaps in existing metrics, particularly with negation and body-part reasoning, and demonstrates that synthetic data can meaningfully tune metrics for better alignment with human judgments. Together, these contributions provide a scalable path to reliable, interpretable T2I metric meta-evaluation and practical metric improvement for real-world generation systems.

Abstract

The assessment of evaluation metrics (meta-evaluation) is crucial for determining the suitability of existing metrics in text-to-image (T2I) generation tasks. Human-based meta-evaluation is costly and time-intensive, and automated alternatives are scarce. We address this gap and propose CROC: a scalable framework for automated Contrastive Robustness Checks that systematically probes and quantifies metric robustness by synthesizing contrastive test cases across a comprehensive taxonomy of image properties. With CROC, we generate a pseudo-labeled dataset (CROC$^{syn}$) of over one million contrastive prompt-image pairs to enable a fine-grained comparison of evaluation metrics. We also use the dataset to train CROCScore, a new metric that achieves state-of-the-art performance among open-source methods, demonstrating an additional key application of our framework. To complement this dataset, we introduce a human-supervised benchmark (CROC$^{hum}$) targeting especially challenging categories. Our results highlight robustness issues in existing metrics: for example, many fail on prompts involving negation, and all tested open-source metrics fail on at least 25% of cases involving correct identification of body parts.

Figures (10)

• Figure 1: Contrastive evaluation of T2I metrics. Given a text-to-image metric that assigns quality scores to a text-image input, matching text-image pairs (green) should receive higher metric scores than non-matching text-image pairs (red). For text-based evaluation, we replace the original text with a contrastive one. Likewise, for image-based evaluation, we replace the original image with a contrastive one. For inverse evaluations, the matching-pair is based on the contrastive text and image that were used in the forward evaluations.
• Figure 2: Top-level properties of our quality taxonomy.
• Figure 3: Selected metric failure-cases CROC$^{hum}$. FC shows a metric that failed on this example and C shows the category of the example. ✓ indicates the matching text-image pair. For text-based evaluation, the metric falsely rates the text with ✗higher than the text with ✓. For image-based evaluation, the metric falsely rates the image with ✗ higher than the image with ✓.
• Figure 4: Scaled image-based accuracy per metric on the top-level properties of CROC$^{syn}$.
• Figure 5: Scaled metric accuracy per category and evaluation direction for CROC$^{hum}$. For 1, a metric correctly rated all matching pairs higher than the contrast, for 0 it is random and for -1 it rated all contrast pairs higher than the matching pair. The tables show the cell-wise average of the forward and inverse evaluations.