QuantuneV2: Compiler-Based Local Metric-Driven Mixed Precision Quantization for Practical Embedded AI Applications

TL;DR

QuantuneV2 addresses the challenge of deploying efficient mixed-precision quantization at compile time for embedded AI by introducing a compiler-based approach that relies on stable local metrics (SQNR delta and MSE) and optimal Graph IR selection. It extends the Glow compiler with calibration, a linear-time sensitivity analysis, and per-layer quantization augmented by operator fusion, achieving $O(n)$ complexity and two inferences to assess impact. Across five models, it reports up to 10.28% accuracy gains and 12.52% faster inference, with substantial reductions in sensitivity-list generation time. The method enables practical deployment of quantized models on diverse hardware (GPU/CPU/ARM) without retraining, significantly reducing compile-time overhead while maintaining or improving accuracy and latency for embedded AI applications.

Abstract

Mixed-precision quantization methods have been proposed to reduce model size while minimizing accuracy degradation. However, existing studies require retraining and do not consider the computational overhead and intermediate representations (IR) generated during the compilation process, limiting their application at the compiler level. This computational overhead refers to the runtime latency caused by frequent quantization and dequantization operations during inference. Performing these operations at the individual operator level causes significant runtime delays. To address these issues, we propose QuantuneV2, a compiler-based mixed-precision quantization method designed for practical embedded AI applications. QuantuneV2 performs inference only twice, once before quantization and once after quantization, and operates with a computational complexity of O(n) that increases linearly with the number of model parameters. We also made the sensitivity analysis more stable by using local metrics like weights, activation values, the Signal to Quantization Noise Ratio, and the Mean Squared Error. We also cut down on computational overhead by choosing the best IR and using operator fusion. Experimental results show that QuantuneV2 achieved up to a 10.28 percent improvement in accuracy and a 12.52 percent increase in speed compared to existing methods across five models: ResNet18v1, ResNet50v1, SqueezeNetv1, VGGNet, and MobileNetv2. This demonstrates that QuantuneV2 enhances model performance while maintaining computational efficiency, making it suitable for deployment in embedded AI environments.

Figures (10)

• Figure 1: Layer-wise sensitivity differences in DNN models
• Figure 2: The runtime of different quantization methods for ResNet18v1. We examine PTQ approaches and illustrate the possibilities of orange square (QAT) methods. The size of the circles represents the model size based on the quantization bit-widths.
• Figure 3: Results of applying mixed-precision to ResNet18v1 and MobileNetv2 across six local metrics (performing dequantization one layer at a time)
• Figure 4: Generation of local metrics according to the stages of operator fusion
• Figure 5: Overview of QuantuneV2