VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model

TL;DR

VARGPT tackles the challenge of unifying visual understanding and generation within a single autoregressive multimodal LLM. It introduces a two-paradigm design: next-token prediction for understanding and next-scale prediction for image generation, implemented within a LLaVA-1.5–based backbone plus a dedicated 2B visual decoder and multi-scale tokenizer. The model is trained in three stages—pretraining, understanding-focused instruction tuning, and generation-focused instruction tuning—with a carefully constructed unified instruction-following dataset. Empirical results show VARGPT surpasses visual-understanding baselines and achieves notable autoregressive visual generation, validating the feasibility and advantages of a fully unified visual autoregressive MLLM. Limitations include image resolution and data quality gaps, with future work pointing toward higher-resolution generation and potential video capabilities.

Abstract

We present VARGPT, a novel multimodal large language model (MLLM) that unifies visual understanding and generation within a single autoregressive framework. VARGPT employs a next-token prediction paradigm for visual understanding and a next-scale prediction paradigm for visual autoregressive generation. VARGPT innovatively extends the LLaVA architecture, achieving efficient scale-wise autoregressive visual generation within MLLMs while seamlessly accommodating mixed-modal input and output within a single model framework. Our VARGPT undergoes a three-stage unified training process on specially curated datasets, comprising a pre-training phase and two mixed visual instruction-tuning phases. The unified training strategy are designed to achieve alignment between visual and textual features, enhance instruction following for both understanding and generation, and improve visual generation quality, respectively. Despite its LLAVA-based architecture for multimodel understanding, VARGPT significantly outperforms LLaVA-1.5 across various vision-centric benchmarks, such as visual question-answering and reasoning tasks. Notably, VARGPT naturally supports capabilities in autoregressive visual generation and instruction-to-image synthesis, showcasing its versatility in both visual understanding and generation tasks. Project page is at: \url{https://vargpt-1.github.io/}

Figures (15)

• Figure 1: A comparative analysis of various MLLMs across multiple visual comprehension and generation benchmarks is presented. The CLIP-scores is employed as a text-to-image visual generation metric, while the remaining metrics are derived from standard visual question-answering benchmarks and multi-modal comprehension benchmarks. Notably, our VARGPT model demonstrates significant superiority over the compared baselines across all comprehension benchmarks. Furthermore, it exhibits exceptional instruction-to-image generation capabilities, thus enhancing its versatility and applicability in diverse visual-linguistic tasks.
• Figure 2: Some generated 256$\times$256 samples by VARGPT trained on ImageNet deng2009imagenet. VARGPT supports text-and-image instructions from user and outputs both text-and-image mixed modal data simultaneously.
• Figure 3: Comparison of different model architectures, where, 'AR' denotes autoregressive, while 'VAR' signifies visual autoregressive. We present a comparative analysis of architectures designed for comprehension-only tasks, generation-only tasks, and unified comprehension and generation, alongside our proposed VARGPT model. Our VARGPT is conceptualized as a purely autoregressive multimodel model, achieving visual comprehension through next-token prediction and visual generation through next-scale prediction paradigms.
• Figure 4: The illustration of the proposed VARGPT framework, which consists of (1) a large language model, visual encoder and a understanding projector for visual understanding; (2) a visual decoder and dual generation projectors for visual generation. VARGPT employs the causal attention in the LLM backbone, while utilizing block causal attention in the visual decoder.
• Figure 5: The three training stages of the VARGPT, including stage-1 pretraining, stage-2 and stage-3 instruction fine-tuning.