Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?

TL;DR

It is observed that while models demonstrate robust textual reasoning, they fail to maintain semantic equivalence when required to render the same results in the image modality, suggesting that the failure stems not from insufficient generation fidelity, but from a breakdown in cross-modal semantic alignment.

Abstract

Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture. However, existing evaluations typically assess these capabilities separately, overlooking semantic equivalence, i.e., the ability to manifest consistent reasoning results regardless of the output modality. In this work, we investigate whether current U-MLLMs satisfy this premise. We observe that while models demonstrate robust textual reasoning, they fail to maintain semantic equivalence when required to render the same results in the image modality. To rigorously diagnose this discrepancy, we introduce VGUBench, a framework to decouple reasoning logic from generation fidelity. VGUBench comprises three diagnostic tasks: (1)Textual Generative Understanding, establishing a baseline for reasoning accuracy in textual response; (2)Visual Generative Understanding, evaluating the ability to generate visual responses that represent the correct answer; and (3)a Visual Rendering control task, which assesses the ability to directly render explicit visual descriptions into images without complex reasoning. Our evaluation reveals a significant disparity: despite strong performance in textual understanding and visual rendering, U-MLLMs exhibit a marked performance collapse when required to generate visual answers to questions. Furthermore, we find a negligible correlation between visual answering performance and basic rendering quality. These results suggest that the failure stems not from insufficient generation fidelity, but from a breakdown in cross-modal semantic alignment. We provide diagnostic insights to address this challenge in future Unified Generation and Understanding Models.

Figures (6)

• Figure 1: Provide U-MLLM with the same question and have it generate answers in text using its understanding capability and answer in image using its generation capability. Ultimately, U-MLLMs exhibited semantic non-equivalence across both output modalities. The images in the "Visual Answer" columns of this figure are all generated by AI models.
• Figure 2: The pipeline of VGUBench construction. First, we randomly sampled and merged the existing 9 text-only question-answer benchmarks to obtain the TGU task test set. Subsequently, we automatically rendered the response texts from the TGU data into a unified format to generate the VGU task test set. Finally, we selected the response texts from TGU, removed duplicates, and applied the same rendering method. We then removed the question texts and used the response texts as input for the Render test set.
• Figure 3: The relationship among the three tasks. In the figure, varying shades of orange represent the three distinct tasks, while dark blue denotes the test data. Black arrows originating from a task module indicate that the data serves as the ground-truth label for evaluation, whereas arrows pointing into a module signify the data as a task input. The VGU is formally similar to the TGU and Render tasks, but it is by no means a simple sum of the two.
• Figure 4: Some inference cases of VGU task. All of this VGU inference sample is "Completeness $<$ 2" and "Correctness $>$ 2". Despite the high scores achieved on the Correctness metric, the inference performance in VGU remains suboptimal. This deficiency is effectively captured and reflected by the lower scores in Completeness.
• Figure 5: Inference Results of Different U-MLLMs on VGUBench. The images in the "VGU" and "Render" columns of this figure are all generated by AI models.