Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

TL;DR

The paper tackles the open question of whether generation can enhance understanding in Unified Multimodal Models (UMMs). It introduces UniMRG, a simple post-training approach that adds four concurrent tasks—image reconstruction, image-to-depth, image-to-segmentation, and image understanding—training with a combined loss $L_{total}$ to instill geometric and structural cues. Across autoregressive, masked autoregressive, and diffusion-based UMMs, UniMRG yields consistent gains in fine-grained perception, hallucination mitigation, spatial understanding, and generation quality, approaching or surpassing state-of-the-art on several benchmarks. Ablation studies show depth and segmentation targets are essential for improving understanding without sacrificing generation, and the method generalizes to out-of-distribution data though its benefits depend on the model’s representational capacity. Overall, UniMRG provides a practical, architecture-agnostic path to more grounded, capable unified multimodal systems and motivates extending intrinsic representations and video settings.

Abstract

Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.

Figures (7)

• Figure 1: Examples of depth and segmentation generation. Off-the-shelf UMMs (e.g., Harmon-1.5B harmon) struggle to generate plausible depth/segmentation maps from input images, often producing outputs closer to RGB reconstruction. UniMRG post-trains UMMs to generate these intrinsic representations, encouraging them to internalize geometric cues (depth) and structural cues (segmentation) that are beneficial for visual understanding.
• Figure 2: Motivation: Intrinsic visual representation generation enhances visual understanding. Top: Off-the-shelf UMMs fail to generate plausible depth maps and struggle with spatial understanding. Bottom: After image-to-depth training, UMMs generate coherent depth maps and exhibit stronger spatial understanding, correctly identifying spatial relationships.
• Figure 3: Overview of UniMRG. The input image is fed into the visual understanding encoder, and the UMM is jointly trained on four tasks: (1) Image reconstruction: reconstructing the input image to enhance generation capabilities. (2) Image-to-depth: generating depth maps to learn geometric cues and spatial relations. (3) Image-to-segmentation: generating segmentation maps to learn structural cues and region partitions. (4) Image understanding: performing standard vision-language understanding tasks. The understanding encoder is updated for UMMs with a shared encoder for generation and understanding; otherwise it is frozen.
• Figure 4: Qualitative results on generation and understanding.
• Figure 5: Qualitative ablation study on the quality of UMM-generated images with different representation generations. D denotes depth representation generation, S denotes segmentation representation generation, and P denotes pixel representation generation.