LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

TL;DR

This work tackles the challenge of automatically evaluating large multimodal image generation (AIGI) in a way that aligns with human preferences. It introduces EvalMi-50K, a large-scale dataset of 50,400 images from 24 T2I models with 2,100 prompts across 20 tasks and over 2 million subjective annotations across perception, text-image correspondence, and task-specific accuracy. Building on this, the authors present LMM4LMM, an all-in-one large multimodal model-based metric that uses instruction tuning and LoRA adaptation to predict text-defined quality levels, provide numerical perception and correspondence scores, and generate task-specific visual QA. Extensive experiments show that LMM4LMM achieves state-of-the-art correlations with human judgments on EvalMi-50K, generalizes well to other benchmarks in zero-shot settings, and benefits from ablations validating the chosen architecture and training strategy. The dataset and metric jointly offer a practical, scalable pathway for evaluating both AIGI generation quality and LMM interpretation capabilities, with public release planned at the project repository.

Abstract

Recent breakthroughs in large multimodal models (LMMs) have significantly advanced both text-to-image (T2I) generation and image-to-text (I2T) interpretation. However, many generated images still suffer from issues related to perceptual quality and text-image alignment. Given the high cost and inefficiency of manual evaluation, an automatic metric that aligns with human preferences is desirable. To this end, we present EvalMi-50K, a comprehensive dataset and benchmark for evaluating large-multimodal image generation, which features (i) comprehensive tasks, encompassing 2,100 extensive prompts across 20 fine-grained task dimensions, and (ii) large-scale human-preference annotations, including 100K mean-opinion scores (MOSs) and 50K question-answering (QA) pairs annotated on 50,400 images generated from 24 T2I models. Based on EvalMi-50K, we propose LMM4LMM, an LMM-based metric for evaluating large multimodal T2I generation from multiple dimensions including perception, text-image correspondence, and task-specific accuracy. Extensive experimental results show that LMM4LMM achieves state-of-the-art performance on EvalMi-50K, and exhibits strong generalization ability on other AI-generated image evaluation benchmark datasets, manifesting the generality of both the EvalMi-50K dataset and LMM4LMM metric. Both EvalMi-50K and LMM4LMM will be released at https://github.com/IntMeGroup/LMM4LMM.

Figures (22)

• Figure 1: We present the large multimodal image generation evaluation database and model, termed EvalMi-50K and LMM4LMM, respectively. (a) We first collect 2100 comprehensive prompts across 20 fine-grained tasks. (b) Then 24 LMM-T2I models are applied to generate 50K images. (c) 100K MOSs and 50K question-answering pairs are acquired from 16 annotators. (d) We design LMM4LMM to evaluate LMM-T2I models. (e) We conduct model comparisons on EvalMi-50K and the other 7 benchmarks.
• Figure 2: (a) Distribution of task counts and scores across different tasks. (b) Distribution of perception and correspondence MOSs.
• Figure 3: Comparison of T2I generation models regarding the perception MOSs, correspondence MOSs, and task-specific accuracy.
• Figure 4: Comparison of MOSs and task-specific accuracy of 24 generation models across 20 tasks with descending order arranged in legend. (a) Results across perception MOSs. (b) Results across correspondence MOSs. (c) Results across task-specific accuracy.
• Figure 5: Overview of the LMM4LMM architecture. The model includes two functions: (1) text-defined quality level and score prediction, (2) task-specific visual question answering. The training process consists of two stages: instruction tuning of the model via text-defined levels, and then fine-tuning the vision encoder and LLM via numerical scores. The model incorporates an image encoder and a text encoder for extracting visual and textual features, which are fed into a pre-trained LLM to generate results. LoRA hulora weights are introduced to the pre-trained image encoder and the LLM to adapt the models to perception quality evaluation and T2I correspondence attribution tasks.