Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis

TL;DR

The paper challenges the assumption that better reconstruction in VQ tokenizers yields better image generation, showing that semantic compression and detail preservation are competing objectives. It introduces SeQ-GAN, a two-phase tokenizer training regime that first promotes semantic compression with a semantic-enhanced perceptual loss and then finetunes a decoder for detail restoration while keeping the encoder/codebook fixed, plus entropy-based codebook usage regularization. Across AR and NAR transformers on unconditional and conditional generation tasks, SeQ-GAN delivers substantial gains in FID and IS, rivalling or surpassing prior VQ-based methods and approaching diffusion/GAN baselines on high-resolution datasets. The authors also provide a visualization pipeline to analyze tokenizer impact on generation and discuss the broader implications for designing discrete latent spaces in visual tokenizers.

Abstract

Vector-Quantized (VQ-based) generative models usually consist of two basic components, i.e., VQ tokenizers and generative transformers. Prior research focuses on improving the reconstruction fidelity of VQ tokenizers but rarely examines how the improvement in reconstruction affects the generation ability of generative transformers. In this paper, we surprisingly find that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation. Instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability to capture textures and structures. We thus highlight two competing objectives of VQ tokenizers for image synthesis: semantic compression and details preservation. Different from previous work that only pursues better details preservation, we propose Semantic-Quantized GAN (SeQ-GAN) with two learning phases to balance the two objectives. In the first phase, we propose a semantic-enhanced perceptual loss for better semantic compression. In the second phase, we fix the encoder and codebook, but enhance and finetune the decoder to achieve better details preservation. The proposed SeQ-GAN greatly improves VQ-based generative models and surpasses the GAN and Diffusion Models on both unconditional and conditional image generation. Our SeQ-GAN (364M) achieves Frechet Inception Distance (FID) of 6.25 and Inception Score (IS) of 140.9 on 256x256 ImageNet generation, a remarkable improvement over VIT-VQGAN (714M), which obtains 11.2 FID and 97.2 IS.

Figures (18)

• Figure 1: Visualizing impact of VQ tokenizers on generative transformers with $\alpha$ trade-off between details preservation and semantic compression in VQ tokenizer training.
• Figure 2: Generation results of SeQ-GAN+NAR. 1st row: LSUN-{cat, bedroom, church}. 2nd row: FFHQ and ImageNet.
• Figure 3: The influence of VQ tokenizers on the training and sampling process of generative transformers.
• Figure 4: Visualization pipeline to examine the influence of VQ tokenizers on generative transformers.
• Figure 5: Visualizing the reconstruction and AR prediction of the baseline tokenizer and its attention-enhanced variant.