Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

TL;DR

The paper investigates how quantization affects bit-level scaling laws in two visual generative paradigms: diffusion-style (continuous latent space) and language-style (discrete tokens). It finds that language-style models consistently exhibit better bit-level scaling, largely due to their discrete representation space, which better tolerates quantization noise. A novel TopKLD distillation method is proposed to balance implicit and explicit knowledge, significantly improving scaling across both integer and floating-point quantization, and enabling low-bit models to approach or surpass their higher-bit counterparts under fixed memory and compute budgets. The work highlights the limited efficacy of existing PTQ methods at very low bits and emphasizes distillation as a powerful approach to recover scaling, offering practical guidance for designing efficient, quantized vision-generative systems.

Abstract

Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing ``implicit knowledge'' and ``explicit knowledge'' during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.

Figures (18)

• Figure 1: Investigation of bit-level scaling laws for VAR (left) and DiT (right) models using standard PTQ and QAT. Left: Quantited VAR exhibits better bit-level scaling laws than full-precision VAR (a shift towards the lower-left region). Right: Quantized DiT shows "almost" no improvement compared to full precision.
• Figure 2: (a) denotes the generation process of visual generative models. A comparison of time-varying errors in quantized DiT (b) and VAR (c) indicates that, despite the errors introduced by quantization during the feature extraction phase in VAR, reconstruction significantly reduces these errors. Conversely, DiT fails to mitigate its errors and experiences an increase, adversely affecting the quality of the final output.
• Figure 3: Analysis of Fault Tolerance in Representation Space Reconstruction Errors. (A lower SNR indicates a higher noise component).
• Figure 4: Visualization of activation values in the 5th, 15th transformer blocks for VAR (top) and DiT (bottom), focusing on the FC1, QKV, and FC2 layers. For additional visualization results, please refer to the appendix \ref{['detailed visualization']}.
• Figure 5: Comparison of bit-level scaling laws across various existing superior PTQ methods. Results show these methods exhibit only marginal improvements at W8A8 and W4A16, and performance significantly deteriorates at lower bit settings, suggesting that existing PTQ fail to substantially enhance bit-level scaling laws.