UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation

TL;DR

UniEval introduces a unified evaluation framework for unified multimodal understanding and generation, eliminating the need for extra models, images, or annotations. It pairs UniBench, a holistic benchmark with 81 fine-grained level-2 tags under 13 level-1 tags, against UniScore, an accuracy-based metric that aggregates case- and tag-level performance across generated images and corresponding questions. Empirical results show UniBench is more challenging than prior benchmarks and UniScore correlates strongly with human judgments, enabling meaningful comparisons across SoTA unified and visual-generation models. The framework provides new insights into the strengths and weaknesses of unified models, highlights the impact of resolution and instruction-following in generation tasks, and sets a new standard for holistic multimodal evaluation with potential extensions to future modalities.

Abstract

The emergence of unified multimodal understanding and generation models is rapidly attracting attention because of their ability to enhance instruction-following capabilities while minimizing model redundancy. However, there is a lack of a unified evaluation framework for these models, which would enable an elegant, simplified, and overall evaluation. Current models conduct evaluations on multiple task-specific benchmarks, but there are significant limitations, such as the lack of overall results, errors from extra evaluation models, reliance on extensive labeled images, benchmarks that lack diversity, and metrics with limited capacity for instruction-following evaluation. To tackle these challenges, we introduce UniEval, the first evaluation framework designed for unified multimodal models without extra models, images, or annotations. This facilitates a simplified and unified evaluation process. The UniEval framework contains a holistic benchmark, UniBench (supports both unified and visual generation models), along with the corresponding UniScore metric. UniBench includes 81 fine-grained tags contributing to high diversity. Experimental results indicate that UniBench is more challenging than existing benchmarks, and UniScore aligns closely with human evaluations, surpassing current metrics. Moreover, we extensively evaluated SoTA unified and visual generation models, uncovering new insights into Univeral's unique values.

Figures (28)

• Figure 1: Overview of UniEval. (a). The proposed UniEval unifies the evaluation of both the multimodal understanding and generation, eliminating limitations due to extra models, labeled images, and the lack of overall results. (b). The proposed UniBench is a holistic and challenging benchmark, with the UniScore metric aligning well with humans.
• Figure 2: Workflow of UniEval. An example in UniBench processed by Janus-Pro-7B chen2025janus to generate four images and outputs choices for each image and question (more examples in Appendix \ref{['app_cases']}). UniScores involves case-level accuracy in a case and tag-level accuracy from answers in the same tag. Our method is versatile, supporting generation evaluation with an extra model, and the understanding via the difference between unified and generation results (see Fig. \ref{['fig_align_combine']} and Appendix \ref{['app_specific_eval']}).
• Figure 3: UniBench. (a). It is built by researchers with LLMs in four steps (details in Appendix \ref{['app_unibench_gen']}). (b). We designed holistic level-1 tags and level-2 tags with many novel attributes. (c). Details of UniBench, including data size, distribution of words and QAs, examples of keywords and prompts (see more keywords in Appendix \ref{['app_keywords']} and prompts in Appendix \ref{['app_unibench_gen']}).
• Figure 4: Correlation with Human Evaluation. x-axis indicates the accuracy for a case from humans, scores in the y-axes are from CLIPScore hessel2021clipscore (text-similarity), VQAScore li2024evaluating (binary option confidence), and the proposed UniScore (multiple options accuracy). Pearson correlation cohen2009pearson measures the normalized covariance, and a higher value indicates closer alignment.
• Figure 5: Task-specific Evaluation. (a): UniEval aligns with the average of MMMU yue2024mmmu and GenEval ghosh2023geneval, exhibiting twice the discriminability measured by the coefficient of variation (CV). (b): UniEval supports task-specific evaluations. "Gen. Results" are evaluated with QWen2.5-VL-7B bai2025qwen2. "Und. Results" are from the difference between Uni. and Gen. results (see Appendix \ref{['app_specific_eval']}), indicating preference on generation (blue region) or Und. in yellow.