GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation

TL;DR

<3-5 sentence high-level summary> This paper tackles the reconstruction–generation trade-off observed when scaling visual tokenizers for autoregressive image generation. It introduces GigaTok, which leverages semantic regularization to align tokenizer features with pre-trained semantic representations, and advocates 1D tokenizers, asymmetric decoder-focused scaling, and entropy loss to stabilize training at billion-scale sizes. By scaling to 3B parameters, GigaTok achieves state-of-the-art reconstruction, generation, and representation results on ImageNet 256×256, outperforming existing discrete-tokenizers and enabling strong downstream AR models. The approach offers a practical path to scalable, high-quality visual tokenization for unified multimodal generation and understanding.

Abstract

In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality -- a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers:(1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to $\bf{3 \space billion}$ parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.

Figures (13)

• Figure 1: Reconstruction vs. generation dilemma: Naively scaling visual tokenizers achieves better reconstruction but degrades downstream autoregressive (AR) generation. In contrast, GigaTok achieves better performance for both reconstruction and generation as tokenizers scale up.
• Figure 2: The 2.9B GigaTok achieves SOTA autoregressive image generation with a 1.4B AR model on ImageNet 256$\times$256 resolution.
• Figure 3: Scaling trend for vanilla 1D tokenizers. As the model size increases, the reconstruction quality of vanilla tokenizers improves but the downstream AR Probing gFID consistently degrades. The increasing AR Probing validation loss indicates that scaling vanilla tokenizers results in a more complex latent space, making it difficult for AR models to learn effectively.
• Figure 4: GigaTok architecture and semantic regularization.Top: We use a hybrid CNN-Transformer design for our visual tokenizer. The transformer layers are implemented with ViT for 2D tokenizer and Q-Former for 1D tokenizer. Bottom: We use a frozen DINOv2 dinov2 image encoder for semantic regularization.
• Figure 5: Training curves for 2.9B XL-XXL tokenizers with and without entropy loss. A 2.9B tokenizer does not converge without entropy loss. The entropy loss encourages high codebook usage and stabilizes training loss.