iFSQ: Improving FSQ for Image Generation with 1 Line of Code

TL;DR

This work addresses the fragmentation between autoregressive (AR) and diffusion image generation by introducing iFSQ, a distribution-aware variant of Finite Scalar Quantization that transforms the latent space into a uniform prior with a simple one-line activation change $y = 2.0 \cdot \sigma(1.6 x) - 1$. iFSQ enables a fair unified tokenizer to benchmark AR and diffusion under identical reconstruction constraints, revealing a sweet spot near $4$ bits per dimension and contrasting scaling: AR offers rapid early convergence while diffusion can achieve a higher performance ceiling with enough compute. The authors extend Representation Alignment to autoregressive models, producing LlamaGen-REPA, which accelerates the emergence of high-level semantics by aligning intermediate layers at roughly one-third of the model depth. Overall, the work provides a practical, plug-and-play tokenizer that unifies discrete and continuous representations and offers actionable insights for model choice, scaling, and alignment in image generation.

Abstract

The field of image generation is currently bifurcated into autoregressive (AR) models operating on discrete tokens and diffusion models utilizing continuous latents. This divide, rooted in the distinction between VQ-VAEs and VAEs, hinders unified modeling and fair benchmarking. Finite Scalar Quantization (FSQ) offers a theoretical bridge, yet vanilla FSQ suffers from a critical flaw: its equal-interval quantization can cause activation collapse. This mismatch forces a trade-off between reconstruction fidelity and information efficiency. In this work, we resolve this dilemma by simply replacing the activation function in original FSQ with a distribution-matching mapping to enforce a uniform prior. Termed iFSQ, this simple strategy requires just one line of code yet mathematically guarantees both optimal bin utilization and reconstruction precision. Leveraging iFSQ as a controlled benchmark, we uncover two key insights: (1) The optimal equilibrium between discrete and continuous representations lies at approximately 4 bits per dimension. (2) Under identical reconstruction constraints, AR models exhibit rapid initial convergence, whereas diffusion models achieve a superior performance ceiling, suggesting that strict sequential ordering may limit the upper bounds of generation quality. Finally, we extend our analysis by adapting Representation Alignment (REPA) to AR models, yielding LlamaGen-REPA. Codes is available at https://github.com/Tencent-Hunyuan/iFSQ

Figures (10)

• Figure 1: Empirical analysis of equal-probability and equal-interval quantization. Since neural network activations naturally follow a gaussian-like distribution lee2017deep, we begin our experiments under this setting. For panels (a)–(f), we quantize the original data into 9 levels, which corresponds to an information entropy of 3.17 bits. We clip the original data to the range $[-3, 3]$ for visualization. Panel (a) shows equal-interval quantization of a standard normal distribution. Panel (b) shows equal-probability quantization of a standard normal distribution. Panel (c) shows equal-interval quantization of a uniform distribution, which is also equal-probability quantization. For panels (a)–(c), the area of each bin represents probability density. In panels (b) and (c), all bins have equal probability, while in panel (a), bins near the mean (0) have higher probability density. Panels (d)–(f) plot the quantization error for each original value (x-axis), where denser regions use larger markers.
• Figure 2: Empirical numerical study of $2.0 \cdot \mathrm{sigmoid}(\alpha x) - 1.0$. Sample 500k points from the standard normal distribution and compute the transformed distribution for several values of $\alpha$. Panel (a) shows the probability density of $2.0 \cdot \mathrm{sigmoid}(\alpha x) - 1.0$ under different $\alpha$ values, and the panel (b) reports the similarity to a uniform distribution measured by KS and RMSE as $\alpha$ varies. Notably, the case with $tanh (\alpha = 2.0)$ corresponds to the original FSQ.
• Figure 3: Effect of $\alpha$ for iFSQ. In (a)–(c), the x-axis denotes $\alpha$. The primary y-axes show PSNR (↑ better), SSIM (↑ better), and LPIPS (↓ better), respectively. The secondary y-axis shows distribution metrics RMSE and KS (both ↓ better). The optimal choice at $\alpha = 1.6$ is highlighted, and tanh performance at $\alpha = 2.0$ is marked, which corresponds to the original FSQ.
• Figure 4: Training Efficiency Comparison: DiT vs LlamaGen (FID vs Compute). At 256 resolution, DiT-Large and LlamaGen-L exhibit approximately 161.04G and 169.65G FLOPs, respectively. Both models employ optimal training configurations derived from ablation studies while sharing the same iFSQ.
• Figure 5: Performance across quantization levels. We plot the performance of iFSQ and AE under different quantization levels, using larger markers to denote models with higher latent dimensionality. Each performance point (including AE) uses a spatial compression factor of 64$\times$. The performance of AE is indicated by horizontal dashed lines, which train under mixed precision and use 16-bit precision to inference.