UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?

TL;DR

This work investigates whether generation can improve understanding in large-scale vision-language models by introducing UniHetero, a concise autoregressive framework that operates on continuous semantic encodings while using a diffusion-based decoder for pixel generation. The key finding is that generation enhances understanding when focused on semantic representations rather than raw pixels, and that data-scale improves this effect; autoregression on the LLM input embedding helps capture visual details with fewer cumulative errors. Pixel-level generation remains challenging, but improvements are attainable through training schedules, masking strategies, and inference-time scaling. Overall, the study validates a data-efficient path to unified VLMs where semantic understanding benefits from generation without sacrificing large-scale understanding capabilities.

Abstract

Vision-language large models are moving toward the unification of visual understanding and visual generation tasks. However, whether generation can enhance understanding is still under-explored on large data scale. In this work, we analysis the unified model with a concise structure, UniHetero, under large-scale pretraining (>200M samples). Our key observations are: (1) Generation can improve understanding, but Only if you generate Semantics, Not Pixels. (2) Generation reveals a superior Data Scaling trend and higher Data Utilization. (3) Autoregression on Input Embedding is effective to capture visual details.

Figures (9)

• Figure 1: The ideal structure of multimodal large language model.
• Figure 2: The unified vision language model structure UniHetero, which enables autoregression on semantic representation in LLM layer and transforms the pixel-level representation in modality related decoder structure.
• Figure 3: Illustration of training and inference progress in UniHetero. (a) shows the training progress with autoregression on input embedding rather than vision encoder and gaussian distribution for mask_rate scheduler. (b) shows the inference progress where a inference-time improvement is proposed to refine the pixel-level generation without influence understanding ability.
• Figure 4: Performance of UniHetero with ploss only under large data-scale. The dashed lines indicates linear regression of data scaling law.
• Figure 5: Ablation performance on image generation strategies. "+ploss" denotes UniHetero with only ploss. "+diffuloss" denotes UniHetero with both ploss and diffuloss. "+warmup" denotes UniHetero with ploss, diffuloss, and a warmed diffusion module.