Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark

Yang Shi, Yuhao Dong, Yue Ding, Yuran Wang, Xuanyu Zhu, Sheng Zhou, Wenting Liu, Haochen Tian, Rundong Wang, Huanqian Wang, Zuyan Liu, Bohan Zeng, Ruizhe Chen, Qixun Wang, Zhuoran Zhang, Xinlong Chen, Chengzhuo Tong, Bozhou Li, Chaoyou Fu, Qiang Liu, Haotian Wang, Wenjing Yang, Yuanxing Zhang, Pengfei Wan, Yi-Fan Zhang, Ziwei Liu

TL;DR

RealUnify presents the first comprehensive benchmark to quantify whether unified multimodal models truly benefit from unification by separately evaluating how understanding can enhance generation (UEG) and how generation can enhance understanding (GEU). It adopts a dual-evaluation protocol (direct and stepwise) across 1,000 tasks in 10 categories, comparing 12 unified models with 6 specialized baselines. Findings show that end-to-end synergy is largely absent, though stepwise decompositions reveal partial capabilities, and an oracle combination demonstrates a higher potential upper bound. The work underscores that architectural unification alone is insufficient and motivates new training strategies and inductive biases to realize genuine cross-capability synergy.

Abstract

The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify, a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation, which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding, which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol, which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling.

RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark

TL;DR

RealUnify presents the first comprehensive benchmark to quantify whether unified multimodal models truly benefit from unification by separately evaluating how understanding can enhance generation (UEG) and how generation can enhance understanding (GEU). It adopts a dual-evaluation protocol (direct and stepwise) across 1,000 tasks in 10 categories, comparing 12 unified models with 6 specialized baselines. Findings show that end-to-end synergy is largely absent, though stepwise decompositions reveal partial capabilities, and an oracle combination demonstrates a higher potential upper bound. The work underscores that architectural unification alone is insufficient and motivates new training strategies and inductive biases to realize genuine cross-capability synergy.

Abstract

The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify, a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation, which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding, which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol, which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling.

Paper Structure

This paper contains 18 sections, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RealUnify
  4. Understanding Enhances Generation (UEG)
  5. Generation Enhances Understanding (GEU)
  6. Benchmark Construction
  7. Experiments
  8. Evaluation on RealUnify
  9. Analysis Experiments
  10. Conclusion
  11. Large Language Model Usage
  12. Benchmark Details
  13. Representative Examples from RealUnify
  14. Task Distribution
  15. Experiment Details
  16. ...and 3 more sections

Figures (12)

  • Figure 1: Illustration of RealUnify. Unlike benchmarks focused on either understanding or generation (Stage 1), those that merely integrate both capabilities (Stage 1.5), or even those that preliminarily explore the mutual enhancement between understanding and generation (Stage 2), RealUnify stands as the first benchmark to comprehensively evaluate and fully harness the synergy between these capabilities, making it a pioneering effort in assessing ability synergy for unified models.
  • Figure 2: Overview of RealUnify. The benchmark includes $2$ main task categories: Understanding Enhances Generation (UEG) and Generation Enhances Understanding (GEU), encompassing $10$ distinct task types. Hints are provided to guide task decomposition in the stepwise evaluation.
  • Figure 3: Illustration of polling evaluation. To assess the accuracy of the generated images, we meticulously design verification questions and employ Gemini 2.5 Pro as the judge in a polling-based evaluation.
  • Figure 4: Statistics of RealUnify. The tasks span $10$ categories, divided into two main groups: UEG and GEU, including $32$ subtasks or areas.
  • Figure 5: Effective examples of stepwise execution in task solving. Through the unified model's inherent understanding and generation abilities, the model is able to implement complex tasks.
  • ...and 7 more figures