Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

TL;DR

GapEval is introduced, a bidirectional benchmark designed to quantify the gap between understanding and generation capabilities, and quantitatively measure the cognitive coherence of the two"unified"directions.

Abstract

Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whether these two capabilities are genuinely aligned and integrated within a single model remains unclear. To investigate this question, we introduce GapEval, a bidirectional benchmark designed to quantify the gap between understanding and generation capabilities, and quantitatively measure the cognitive coherence of the two "unified" directions. Each question can be answered in both modalities (image and text), enabling a symmetric evaluation of a model's bidirectional inference capability and cross-modal consistency. Experiments reveal a persistent gap between the two directions across a wide range of UMMs with different architectures, suggesting that current models achieve only surface-level unification rather than deep cognitive convergence of the two. To further explore the underlying mechanism, we conduct an empirical study from the perspective of knowledge manipulation to illustrate the underlying limitations. Our findings indicate that knowledge within UMMs often remains disjoint. The capability emergence and knowledge across modalities are unsynchronized, paving the way for further exploration.

Figures (10)

• Figure 1: Overview of GapEval. (a) We present a high-quality bidirectional benchmark specifically designed for UMMs to evaluate and qualify the inherent gap between understanding and generation. (b) Our experiments extend 9 UMMs across various architectures, revealing the significant gap between the two capabilities. (c) We further conduct an in-depth empirical study from a perspective of knowledge manipulation, highlighting the significant gap in the knowledge level.
• Figure 2: Illustrative examples from GapEval. The generated texts and images are shown with the ground truth (texts, images or both). GPT Family denotes the GPT5-mini (for text generation), and the GPT-Image-1 (for image generation).
• Figure 3: Gap score heatmap over understanding and generation performance. (Und., Gen., Gap Score) data points are plotted on the heatmap, reflecting the relation between three dimensions. The trend curve demonstrates the model performance distribution.
• Figure 4: Training data case gallery.
• Figure 5: Performance increasing over the training steps on the knowledge edit task. The figure has also exhibited the output of models from different training stages. The knowledge entity involved here is (Microwave Oven->Rice Cooker).