Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction

TL;DR

This work tackles the challenge of autoregressive image generation with large VQ-VAE codebooks by identifying redundancy within the token vocabulary. It introduces a coarse-to-fine (CTF) framework that clusters codewords into a smaller set of coarse labels and then predicts fine-grained labels conditioned on these coarse labels, using a two-stage generation process. Empirically, CTF yields significant improvements in Inception Score (average +59 points) and competitive or improved FID (up to -1), while also delivering faster sampling due to a reduced autoregressive vocabulary and parallel fine-label prediction. The approach is architecture-agnostic, scales with model size, and accelerates training convergence, making high-quality autoregressive image generation with large codebooks both effective and efficient.

Abstract

Autoregressive models have shown remarkable success in image generation by adapting sequential prediction techniques from language modeling. However, applying these approaches to images requires discretizing continuous pixel data through vector quantization methods like VQ-VAE. To alleviate the quantization errors that existed in VQ-VAE, recent works tend to use larger codebooks. However, this will accordingly expand vocabulary size, complicating the autoregressive modeling task. This paper aims to find a way to enjoy the benefits of large codebooks without making autoregressive modeling more difficult. Through empirical investigation, we discover that tokens with similar codeword representations produce similar effects on the final generated image, revealing significant redundancy in large codebooks. Based on this insight, we propose to predict tokens from coarse to fine (CTF), realized by assigning the same coarse label for similar tokens. Our framework consists of two stages: (1) an autoregressive model that sequentially predicts coarse labels for each token in the sequence, and (2) an auxiliary model that simultaneously predicts fine-grained labels for all tokens conditioned on their coarse labels. Experiments on ImageNet demonstrate our method's superior performance, achieving an average improvement of 59 points in Inception Score compared to baselines. Notably, despite adding an inference step, our approach achieves faster sampling speeds.

Figures (3)

• Figure 1: (a) The codeword clustering process, where token indices are grouped based on the similarity of their corresponding feature vectors in the codebook. (b) Visual demonstration of token redundancy: replacing each token with another randomly sampled from the same cluster produces images with only minor variations in detail, preserving the overall structure and content. (c) Illustration of our two-stage generation process: in the first stage, the model autoregressively predicts coarse labels (cluster indices) for each token in the sequence; then the second stage model predicts fine labels (indices in the codebook) for all tokens in a single step.
• Figure 2: Model performance comparison on different epochs. When our method is applied, models achieve significantly better performance.
• Figure 3: Generation results of our method (based on LlamaGen-XL) on ImageNet 256×256 benchmark.