Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark

TL;DR

Experimental results demonstrate that the distilled open-source MLLM significantly outperforms the current state-of-the-art GPT-4o-base baseline, VIEScore, with over 4.6\% improvement in Spearman and Kendall correlations with human judgments.

Abstract

Driven by the remarkable progress in diffusion models, text-to-image generation has made significant strides, creating a pressing demand for automatic quality evaluation of generated images. Current state-of-the-art automatic evaluation methods heavily rely on Multi-modal Large Language Models (MLLMs), particularly powerful commercial models like GPT-4o. While these models are highly effective, their substantial costs limit scalability in large-scale evaluations. Adopting open-source MLLMs is an alternative; however, their performance falls short due to significant limitations in processing multi-modal data compared to commercial MLLMs. To tackle these problems, we first propose a task decomposition evaluation framework based on GPT-4o to automatically construct a new training dataset, where the complex evaluation task is decoupled into simpler sub-tasks, effectively reducing the learning complexity. Based on this dataset, we design innovative training strategies to effectively distill GPT-4o's evaluation capabilities into a 7B open-source MLLM, MiniCPM-V-2.6. Furthermore, to reliably and comprehensively assess prior works and our proposed model, we manually annotate a meta-evaluation benchmark that includes chain-of-thought explanations alongside quality scores for generated images. Experimental results demonstrate that our distilled open-source MLLM significantly outperforms the current state-of-the-art GPT-4o-base baseline, VIEScore, with over 4.6\% improvement in Spearman and Kendall correlations with human judgments.

Figures (17)

• Figure 1: The overview of our proposed Task Decomposition Evaluation Framework, consisting of three steps: (1) Evaluation Content Extraction and Question Generation; (2) Caption and Answer Generation; (3) Explanation and Scoring.
• Figure 2: Performance of MiniCPM-V-2.6 and GPT-4o on text-to-image evaluation. Self EC and GPT-4o EC represent the model uses evaluation content extracted by itself and GPT-4o, respectively. Greater values of $\rho$ and $\tau$ indicates better performance.
• Figure 3: A bad case of evaluation content extraction step by MiniCPM-V-2.6 without fine-tuning.
• Figure 4: The relationship between task decomposition evaluation framework and fine-tuning sub-tasks.
• Figure 5: Improved results in fine-tuned MLLMs over base models' zero-shot results. $\rho,\tau$ are the correlation scores with human judgments. Red arrows show improvement ratio.