Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better

TL;DR

This work tackles the limitation of vision-language models that supervise only textual outputs by introducing Autoregressive Semantic Visual Reconstruction (ASVR), a unified autoregressive framework that learns from semantic visual tokens. It uses a semantic visual tokenizer and a continuous visual encoder to enable autoregressive prediction of visual semantics alongside text, showing that predicting appearance alone can hinder performance while semantic reconstruction consistently improves multimodal understanding. Across diverse data scales and backbones, ASVR yields significant gains on 14 benchmarks, demonstrating robustness to input type (continuous vs discrete features) and high-resolution settings. The results suggest a scalable, unified training paradigm for LVLMs that foregrounds visual perception and semantic grounding, with potential future extensions to include image generation within the same autoregressive framework.

Abstract

Typical large vision-language models (LVLMs) apply autoregressive supervision solely to textual sequences, without fully incorporating the visual modality into the learning process. This results in three key limitations: (1) an inability to utilize images without accompanying captions, (2) the risk that captions omit critical visual details, and (3) the challenge that certain vision-centric content cannot be adequately conveyed through text. As a result, current LVLMs often prioritize vision-to-language alignment while potentially overlooking fine-grained visual information. While some prior works have explored autoregressive image generation, effectively leveraging autoregressive visual supervision to enhance image understanding remains an open challenge. In this paper, we introduce Autoregressive Semantic Visual Reconstruction (ASVR), which enables joint learning of visual and textual modalities within a unified autoregressive framework. We show that autoregressively reconstructing the raw visual appearance of images does not enhance and may even impair multimodal understanding. In contrast, autoregressively reconstructing the semantic representation of images consistently improves comprehension. Notably, we find that even when models are given continuous image features as input, they can effectively reconstruct discrete semantic tokens, resulting in stable and consistent improvements across a wide range of multimodal understanding benchmarks. Our approach delivers significant performance gains across varying data scales (556k-2M) and types of LLM bacbones. Specifically, ASVR improves LLaVA-1.5 by 5% in average scores across 14 multimodal benchmarks. The code is available at https://github.com/AlenjandroWang/ASVR.

Figures (3)

• Figure 1: (Left) A simple illustration that reflects the information loss faced by language-centric approaches. (Right) Our proposed Autoregressive Semantic Visual Reconstruction (ASVR) brings significant improvements across various aspects, including General VQA, Visual-centric, Hallucination, and OCR. All the scores are normalized by $x_{\text{norm}}=(x-x_{\text{min}}+10)/(x_{\text{max}}-x_{\text{min}}+10)$.
• Figure 2: Left: the typical LVLM framework exemplified by LLaVA liu2023llava. Right: overview of ASVR's model architecture and training procedure. The input image and its corresponding text are tokenized into sequences of discrete token indices for unified autoregressive supervision over both visual and textual outputs. For each module, the icon before the slash indicates whether it is frozen or tunable during pre-training, while the icon after the slash indicates its configuration during instruction tuning. "s" and "e" denote the start and end of the text tokens, respectively.
• Figure 3: Qualitative comparison on attention maps, where we keep the same LLM and training data. With extra vision-centric supervision signals, ROSS urges the model to focus on specific image contents corresponding to the question with higher attention values.