Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UEval: A Benchmark for Unified Multimodal Generation

Bo Li, Yida Yin, Wenhao Chai, Xingyu Fu, Zhuang Liu

TL;DR

UEval introduces a scalable, rubric-based benchmark for unified multimodal generation, requiring models to produce both images and text in response to 1,000 questions drawn from 8 real-world tasks. A data-dependent rubric generation pipeline uses a frontier MLLM to craft evaluation criteria, which are refined by human reviewers, enabling fine-grained automatic scoring across 10,417 criteria. Empirical results show current unified models face substantial challenges, with strongest frontier models only modestly outperforming open-source ones, and reasoning-enabled models delivering notable gains, suggesting reasoning traces can enhance multimodal generation. The work provides a rigorous, interpretable framework for assessing and guiding progress toward truly unified vision-language generation systems with practical implications for real-world multimodal AI applications.

Abstract

We introduce UEval, a benchmark to evaluate unified models, i.e., models capable of generating both images and text. UEval comprises 1,000 expert-curated questions that require both images and text in the model output, sourced from 8 real-world tasks. Our curated questions cover a wide range of reasoning types, from step-by-step guides to textbook explanations. Evaluating open-ended multimodal generation is non-trivial, as simple LLM-as-a-judge methods can miss the subtleties. Different from previous works that rely on multimodal Large Language Models (MLLMs) to rate image quality or text accuracy, we design a rubric-based scoring system in UEval. For each question, reference images and text answers are provided to a MLLM to generate an initial rubric, consisting of multiple evaluation criteria, and human experts then refine and validate these rubrics. In total, UEval contains 10,417 validated rubric criteria, enabling scalable and fine-grained automatic scoring. UEval is challenging for current unified models: GPT-5-Thinking scores only 66.4 out of 100, while the best open-source model reaches merely 49.1. We observe that reasoning models often outperform non-reasoning ones, and transferring reasoning traces from a reasoning model to a non-reasoning model significantly narrows the gap. This suggests that reasoning may be important for tasks requiring complex multimodal understanding and generation.

UEval: A Benchmark for Unified Multimodal Generation

TL;DR

UEval introduces a scalable, rubric-based benchmark for unified multimodal generation, requiring models to produce both images and text in response to 1,000 questions drawn from 8 real-world tasks. A data-dependent rubric generation pipeline uses a frontier MLLM to craft evaluation criteria, which are refined by human reviewers, enabling fine-grained automatic scoring across 10,417 criteria. Empirical results show current unified models face substantial challenges, with strongest frontier models only modestly outperforming open-source ones, and reasoning-enabled models delivering notable gains, suggesting reasoning traces can enhance multimodal generation. The work provides a rigorous, interpretable framework for assessing and guiding progress toward truly unified vision-language generation systems with practical implications for real-world multimodal AI applications.

Abstract

We introduce UEval, a benchmark to evaluate unified models, i.e., models capable of generating both images and text. UEval comprises 1,000 expert-curated questions that require both images and text in the model output, sourced from 8 real-world tasks. Our curated questions cover a wide range of reasoning types, from step-by-step guides to textbook explanations. Evaluating open-ended multimodal generation is non-trivial, as simple LLM-as-a-judge methods can miss the subtleties. Different from previous works that rely on multimodal Large Language Models (MLLMs) to rate image quality or text accuracy, we design a rubric-based scoring system in UEval. For each question, reference images and text answers are provided to a MLLM to generate an initial rubric, consisting of multiple evaluation criteria, and human experts then refine and validate these rubrics. In total, UEval contains 10,417 validated rubric criteria, enabling scalable and fine-grained automatic scoring. UEval is challenging for current unified models: GPT-5-Thinking scores only 66.4 out of 100, while the best open-source model reaches merely 49.1. We observe that reasoning models often outperform non-reasoning ones, and transferring reasoning traces from a reasoning model to a non-reasoning model significantly narrows the gap. This suggests that reasoning may be important for tasks requiring complex multimodal understanding and generation.
Paper Structure (31 sections, 15 figures, 5 tables)

This paper contains 31 sections, 15 figures, 5 tables.

Table of Contents

  1. Introduction
  2. UEval
  3. Dataset Composition
  4. Space
  5. Textbook
  6. Diagram
  7. Paper
  8. Guide
  9. Rubric Generation and Evaluation
  10. Rubric generation
  11. Human review
  12. Rubric-based evaluation
  13. Experiments
  14. Settings
  15. Models
  16. ...and 16 more sections

Figures (15)

  • Figure 1: Put here the caption for your figure. Place your figure caption in this section. Your figure description goes here. Insert the caption for your figure here. Add caption here. Include the description of your figure here. Place additional details about the figure content in this area. Describe the visual elements shown in the figure.
  • Figure 2: Left: Previous unified model evaluations focus on either image understanding (i.e., VQA) or image generation from captions (i.e., T2I). In contrast, UEval requires models to reason across modalities and generate responses in both images and text. The VQA example is from BLINK fu2024blink, and the T2I example is from ScienceT2I li2025science. Right: The chart illustrates the number of questions and rubric criteria across tasks in UEval.
  • Figure 3: We visualize images generated by GPT-5-Thinking, Gemini-2.5-Flash, and Emu3.5 cui2025emu35nativemultimodalmodels. These images fail to answer the questions accurately. For example, Gemini-2.5-Flash depicts a nonexistent external platform above the Statue of Liberty instead of the crown interior.
  • Figure 4: We prompt GPT-5-Thinking, Gemini-2.5-Flash, and Emu3.5 to synthesize step-by-step visual guides for each task. The generated images often exhibit temporal inconsistencies. For instance, in the art task, when drawing a cartoon cat, GPT-5-Thinking mislabels sub-images (e.g., two images tagged as step 5). For visualization, we stack the images generated by Gemini-2.5-Flash into a single grid.
  • Figure 5: We propose using data-dependent rubrics to evaluate outputs from unified models. For each question, a model drafts an itemized rubric based on the question and the reference image-text pair. A judge model then scores the generated response against each rubric criterion.
  • ...and 10 more figures