UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning

TL;DR

UGen addresses the challenge of a single autoregressive model performing text processing, image understanding, and image generation by unifying tokenization and prompting within one transformer. It introduces progressive vocabulary learning to gradually activate visual tokens during pretraining, reducing cross-modal interference and enhancing deep fusion across modalities. Empirical results show a 13.3% improvement over vanilla unified autoregressive methods and competitive performance against task-specific models, with qualitative evidence of mixed-modality generation. This work demonstrates that a simple autoregressive architecture, equipped with progressive visual vocabulary integration, can achieve strong, versatile multimodal capabilities and sets a path toward scalable, unified multimodal pretraining.

Abstract

We introduce UGen, a unified autoregressive multimodal model that demonstrates strong performance across text processing, image understanding, and image generation tasks simultaneously. UGen converts both texts and images into discrete token sequences and utilizes a single transformer to generate them uniformly in an autoregressive manner. To address the challenges associated with unified multimodal learning, UGen is trained using a novel mechanism, namely progressive vocabulary learning. In this process, visual token IDs are incrementally activated and integrated into the training phase, ultimately enhancing the effectiveness of unified multimodal learning. Experiments on comprehensive text and image tasks show that UGen achieves a significant overall performance improvement of 13.3% compared to the vanilla unified autoregressive method, and it also delivers competitive results across all tasks against several task-specific models.

Figures (9)

• Figure 1: The performance comparison among task-specific autoregressive models (Task-specific AR), current vanilla unified autoregressive model (Vanilla Unified AR) and UGen. Specifically, Text, Image-Und, Image-Gen denote text processing, image understanding and image generation tasks. Task-specific AR models are separated autoregressive models trained with single task data respectively. Vanilla Unified AR model is current unified autoregressive model trained with traditional joint learning approach wu2024liquid.
• Figure 2: An Overview of UGen. Left: A unified autoregressive generative architecture for language-vision understanding and generation. Both texts and images are converted into discrete token sequences and a single transformer is applied to generate them uniformly in an autoregressive manner. Right: The illustration of progressive vocabulary learning, which the visual token IDs are gradually activated and integrated into the training process.
• Figure 3: Illustration of unified prompting.
• Figure 4: The perplexity trajectories of vanilla unified autoregressive models with different sized visual vocabularies.
• Figure 5: The Qualitative results of UGen