Product-Quantised Image Representation for High-Quality Image Synthesis

TL;DR

PQGAN introduces product quantisation for latent image representations within the VQGAN framework, achieving superior reconstruction fidelity and robustness compared to continuous baselines. By factorising latent codes into $S$ subspaces with $K$ codewords per subspace, it creates a fictive codebook of size $K^S$ while keeping per-subspace complexity manageable. The approach yields state-of-the-art PSNRs up to $37.4$ dB and substantial improvements in FID, LPIPS, and CMMD, and it can be integrated with pre-trained diffusion models to enable faster generation or higher-resolution outputs. Across ImageNet and transfer to FFHQ/LSUN, PQGAN demonstrates strong generalisation and provides a practical, scalable path for discrete latent representations in large-scale image synthesis.

Abstract

Product quantisation (PQ) is a classical method for scalable vector encoding, yet it has seen limited usage for latent representations in high-fidelity image generation. In this work, we introduce PQGAN, a quantised image autoencoder that integrates PQ into the well-known vector quantisation (VQ) framework of VQGAN. PQGAN achieves a noticeable improvement over state-of-the-art methods in terms of reconstruction performance, including both quantisation methods and their continuous counterparts. We achieve a PSNR score of 37dB, where prior work achieves 27dB, and are able to reduce the FID, LPIPS, and CMMD score by up to 96%. Our key to success is a thorough analysis of the interaction between codebook size, embedding dimensionality, and subspace factorisation, with vector and scalar quantisation as special cases. We obtain novel findings, such that the performance of VQ and PQ behaves in opposite ways when scaling the embedding dimension. Furthermore, our analysis shows performance trends for PQ that help guide optimal hyperparameter selection. Finally, we demonstrate that PQGAN can be seamlessly integrated into pre-trained diffusion models. This enables either a significantly faster and more compute-efficient generation, or a doubling of the output resolution at no additional cost, positioning PQ as a strong extension for discrete latent representation in image synthesis.

Figures (10)

• Figure 1: Reconstruction results for our quantised PQGAN vs. the continuous VAE in StableDiffusion2.1 rombach2022high. Even with stronger spatial downsizing factor $F$ for the latent representation, our product quantised representation space shows higher fidelity (see details in faces).
• Figure 2: Comparison of vector quantisation (VQ, top) and product quantisation (PQ, bottom) in the latent space of an autoencoder. VQ replaces each latent vector with the nearest entry from a single codebook of size $K$. PQ splits the vector into $S$ subspaces, quantising each independently, resulting in a combinatorially large fictive codebook of size $K^S$ while maintaining low per-subspace complexity.
• Figure 3: Quantitative analysis of product quantisation configurations. (a) Performance across latent dimensionality $d$, number of subspaces $S$, and codebook size $K$ as measured by FID. (b) Codebook utilisation metrics (entropy and perplexity) for the same configurations, showing improved usage and efficiency with increasing factorisation.
• Figure 4: Generated examples for our PQSD-HR (left), PQSD-Precise (top), PQSD-Quick (bottom).
• Figure 5: Reconstruction time and quantisation overhead comparison for product quantisation configurations.