VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation

TL;DR

VTBench tackles the gap in autoregressive image generation by isolating and evaluating visual tokenizers as standalone components across reconstruction, detail, and text fidelity. It reveals a substantial performance gap between discrete VTs and continuous VAEs, with discrete methods struggling under arbitrary resolutions and multilingual text, underscoring the bottleneck in tokenization. The paper introduces three tasks and a diverse dataset suite with OCR-based metrics to diagnose VT performance and drive open-source VT development. It also analyzes GPT-4o's image generation to contextualize VT capabilities within language-grounded multimodal generation. Overall, VTBench offers a practical, scalable benchmark to steer progress toward flexible, high-quality visual tokenizers compatible with LLM-driven multimodal systems.

Abstract

Autoregressive (AR) models have recently shown strong performance in image generation, where a critical component is the visual tokenizer (VT) that maps continuous pixel inputs to discrete token sequences. The quality of the VT largely defines the upper bound of AR model performance. However, current discrete VTs fall significantly behind continuous variational autoencoders (VAEs), leading to degraded image reconstructions and poor preservation of details and text. Existing benchmarks focus on end-to-end generation quality, without isolating VT performance. To address this gap, we introduce VTBench, a comprehensive benchmark that systematically evaluates VTs across three core tasks: Image Reconstruction, Detail Preservation, and Text Preservation, and covers a diverse range of evaluation scenarios. We systematically assess state-of-the-art VTs using a set of metrics to evaluate the quality of reconstructed images. Our findings reveal that continuous VAEs produce superior visual representations compared to discrete VTs, particularly in retaining spatial structure and semantic detail. In contrast, the degraded representations produced by discrete VTs often lead to distorted reconstructions, loss of fine-grained textures, and failures in preserving text and object integrity. Furthermore, we conduct experiments on GPT-4o image generation and discuss its potential AR nature, offering new insights into the role of visual tokenization. We release our benchmark and codebase publicly to support further research and call on the community to develop strong, general-purpose open-source VTs.

Figures (14)

• Figure 1: Image generation and reconstruction across different models. (Top) Images generated from prompts using various models. (Bottom) Reconstructions of the GPT-4o-generated chick image using VTs from different models. PSNR $\uparrow$ values (shown in white) indicates reconstruction fidelity.
• Figure 2: Overview of visual tokenizer architectures and integration with AR image generation.
• Figure 3: Overview of VTBench construction. (a) VTBench consists of three core tasks for evaluating visual tokenizers. (b) Evaluation metrics include both image quality metrics and text-specific metrics.
• Figure 4: Examples of task 1: (1) ImageNet, (2) High Resolution and (3) Varying Resolution. FlowMo Hi produces corrupted images in High Resolution and Varying Resolution, while Janus Pro, LlamaGen, and Chameleon generate images with incorrect resolution and distorted semantic content.
• Figure 5: Visualize qualitative results of detail preservation and text preservation.