ISQuant: apply squant to the real deployment
Dezan Zhao
TL;DR
ISQuant tackles the deployment gap between fake quantization and real-world use by offering a data-free, fast 8-bit deployment method built on SQuant. It switches to per-tensor symmetric quantization, folds BatchNorm into convolution, and uses integer arithmetic to avoid training data and extra computation. On ImageNet, ISQuant achieves negligible accuracy loss (around 1% or less) on large models at 8-bit, with acceptable performance at 7-bit for many architectures, demonstrating practical deployment benefits. The work highlights a scalable path to deploying high-performance quantized models without data and with minimal overhead.
Abstract
The model quantization technique of deep neural networks has garnered significant attention and has proven to be highly useful in compressing model size, reducing computation costs, and accelerating inference. Many researchers employ fake quantization for analyzing or training the quantization process. However, fake quantization is not the final form for deployment, and there exists a gap between the academic setting and real-world deployment. Additionally, the inclusion of additional computation with scale and zero-point makes deployment a challenging task. In this study, we first analyze why the combination of quantization and dequantization is used to train the model and draw the conclusion that fake quantization research is reasonable due to the disappearance of weight gradients and the ability to approximate between fake and real quantization. Secondly, we propose ISQuant as a solution for deploying 8-bit models. ISQuant is fast and easy to use for most 8-bit models, requiring fewer parameters and less computation. ISQuant also inherits the advantages of SQuant, such as not requiring training data and being very fast at the first level of quantization. Finally We conduct some experiments and found the results is acceptable.our code is available at https://github.com/