Scaling Image Tokenizers with Grouped Spherical Quantization

TL;DR

This work identifies critical bottlenecks in scalable image tokenizers, notably reliance on GAN hyperparameters and biased benchmarks. It introduces Grouped Spherical Quantization (GSQ), combining spherical codebook initialization, lookup normalization, and latent decomposition to enable scalable, high-fidelity tokenizers. Through exhaustive VAE and GAN experiments, the authors show that GSQ yields superior reconstruction with far fewer training steps, and that scaling the latent space via grouping and expanding the codebook significantly improves performance, even at aggressive spatial downsampling up to 16×. The findings establish GSQ as a scalable approach for discrete image representations, with clear guidance on architecture, training, and regularization, and demonstrate practical gains for downstream generative tasks.

Abstract

Vision tokenizers have gained a lot of attraction due to their scalability and compactness; previous works depend on old-school GAN-based hyperparameters, biased comparisons, and a lack of comprehensive analysis of the scaling behaviours. To tackle those issues, we introduce Grouped Spherical Quantization (GSQ), featuring spherical codebook initialization and lookup regularization to constrain codebook latent to a spherical surface. Our empirical analysis of image tokenizer training strategies demonstrates that GSQ-GAN achieves superior reconstruction quality over state-of-the-art methods with fewer training iterations, providing a solid foundation for scaling studies. Building on this, we systematically examine the scaling behaviours of GSQ, specifically in latent dimensionality, codebook size, and compression ratios, and their impact on model performance. Our findings reveal distinct behaviours at high and low spatial compression levels, underscoring challenges in representing high-dimensional latent spaces. We show that GSQ can restructure high-dimensional latent into compact, low-dimensional spaces, thus enabling efficient scaling with improved quality. As a result, GSQ-GAN achieves a 16x down-sampling with a reconstruction FID (rFID) of 0.50.

Figures (13)

• Figure 1: Reconstruction performance of GSQ with a latent dimension of 16 at 16$\times$ spatial compression, compared to the state-of-the-art.
• Figure 2: Scaling behaviour of the latent dimension v.s. spatial compression factor in GSQ; $d=16$ is fixed while groups $G$ increase to expand latent space.
• Figure 4: Comparisons of quantizers for VAE-F8 training. VQ is initialized with uniform distribution; all models have the same backbone, latent dimension, and vocabulary size.
• Figure 5: GSQ-GAN ablations on wider and deeper networks w/ and w/o attention blocks. Models are trained on $256^2$ resolution on ImageNet.
• Figure 6: Scaling of latent dimension and vocabulary size for GSQ at 8$\times$ spatial compression.