Multi-Modal Language Models as Text-to-Image Model Evaluators

TL;DR

MT2IE introduces a dynamic, multimodal-LM–driven framework for evaluating text-to-image models by having evaluator agents generate and score prompts and image generations. It demonstrates that open-source MLLMs can match or exceed human correlation on static benchmarks, generate efficient iterative benchmarks, and adapt prompts to model performance while preserving model rankings. The approach significantly reduces the number of prompts required (down to about 20) compared with traditional benchmarks (e.g., 1600 prompts), and generalizes to other axes such as image aesthetics. This dynamic, interactive evaluation paradigm improves robustness against benchmark saturation and data-deprecation while enabling bespoke, model-specific assessments with practical implications for T2I development and benchmarking.

Abstract

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate the T2I progress. In this paper, we explore the potential of multi-modal large language models (MLLMs) as evaluator agents that interact with a T2I model, with the objective of assessing prompt-generation consistency and image aesthetics. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework that iteratively generates prompts for evaluation, scores generated images and matches T2I evaluation of existing benchmarks with a fraction of the prompts used in existing static benchmarks. Moreover, we show that MT2IE's prompt-generation consistency scores have higher correlation with human judgment than scores previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance, producing the same relative T2I model rankings as existing benchmarks while using only 1/80th the number of prompts for evaluation.

Figures (22)

• Figure 1: We present Multimodal Text-to-Image Eval (MT2IE), our novel evaluation methodology for text-to-image (T2I) generation. Previous approaches to T2I model evaluation require large, static benchmarks and often multiple external models. Our approach in a single system that both generates and scores prompts. We show that MT2IE can generate as few as 20 prompts while still maintaining consistent model rankings of other approaches that use more compute and more data.
• Figure 2: An example our method MT2IE generating progressively difficult prompts, with additions shown in purple. Corresponding generated images and scores are also shown. Prompts become more complex while maintaining linguistic structure.
• Figure 3: Results for MT2IE, where the MLLM iteratively generates progressively more difficult prompts over 5 iterations.
• Figure 4: Model rankings produced by MT2IE closely match GenAIBench's ranking, while only using 20 generated prompts. Other methods produce mismatched model rankings when using 20 sampled GenAIBench prompts.
• Figure 5: Example run with MT2IE's Adaptive Evaluation, with generated images and alignment scores to illustrate the evaluation process. Portions in purple were added to the prompt from the previous iteration; portions in red were removed. Generations have high scores for iterations 1-3, so the prompts get progressively more difficult. When the score drops significantly at iteration 4, MT2IE reacts and simplifies the prompt.