Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling

TL;DR

The paper tackles the substantial training overhead of multi-bit quantization networks by introducing two complementary techniques: weight bias correction to align quantized weights and enable shared batch normalization across bit-widths, and bit-wise coreset sampling to tailor informative data subsets per bit-width with gradient-based importance scores. Weight bias correction reduces calibration needs, while bit-wise coresets exploit cross-bitwidth implicit knowledge transfer to cut data usage and computation. Experiments across CIFAR-10/100, TinyImageNet, and ImageNet-1K with ResNet and ViT architectures demonstrate competitive accuracy with up to 7.88× faster training, and ablations show the crucial roles of BN adaptation and per-bit-width scoring. Together, these methods offer a scalable path to efficient, flexible multi-bit quantization applicable to diverse architectures and tasks.

Abstract

Multi-bit quantization networks enable flexible deployment of deep neural networks by supporting multiple precision levels within a single model. However, existing approaches suffer from significant training overhead as full-dataset updates are repeated for each supported bit-width, resulting in a cost that scales linearly with the number of precisions. Additionally, extra fine-tuning stages are often required to support additional or intermediate precision options, further compounding the overall training burden. To address this issue, we propose two techniques that greatly reduce the training overhead without compromising model utility: (i) Weight bias correction enables shared batch normalization and eliminates the need for fine-tuning by neutralizing quantization-induced bias across bit-widths and aligning activation distributions; and (ii) Bit-wise coreset sampling strategy allows each child model to train on a compact, informative subset selected via gradient-based importance scores by exploiting the implicit knowledge transfer phenomenon. Experiments on CIFAR-10/100, TinyImageNet, and ImageNet-1K with both ResNet and ViT architectures demonstrate that our method achieves competitive or superior accuracy while reducing training time up to 7.88x. Our code is released at https://github.com/a2jinhee/EMQNet_jk.

Figures (9)

• Figure 1: A conceptual diagram of i) Dedicated, ii) Any-Precision, and iii) our training pipelines. $\mathcal{D}$ and $\mathcal{S}$ indicate the full training dataset and coreset, respectively.
• Figure 2: A summary of Any-Precision's and our training pipeline.
• Figure 3: (a) Mismatch in activation distributions between different bit-widths, and (b) variance ratio between quantized weights and original weights in ResNet-50.
• Figure 4: Angle between the 8-bit and 2-bit gradients across layers.
• Figure 5: Spearman correlation between ranks at different epochs.