SplitQuant: Layer Splitting for Low-Bit Neural Network Quantization

TL;DR

Quantization of DNNs is essential for reducing memory and speeding inference but is vulnerable to outliers that widen the value range and degrade resolution at low bit-widths. SplitQuant addresses this by restructuring each quantizable layer into three equivalent layers with separate scaling factors, clustering weights and biases into three groups via $k=3$-means, and splitting activations when feasible, thereby narrowing per-layer ranges and boosting quantization accuracy. Empirical results on two fine-tuned BERT-Tiny models show INT2 improvements of 3.3 and 2.1 percentage points toward FP32 baselines, with additional gains at higher bit-widths, demonstrating the method’s effectiveness as a complementary quantization preprocessing step. The approach is practical for TinyML and edge AI, can integrate with existing quantization pipelines, and opens avenues for application to larger models and sparse inference workflows.

Abstract

Quantization for deep neural networks (DNNs) is the process of mapping the parameter values of DNNs from original data types to other data types of lower precision to reduce model sizes and make inference faster. Quantization often maps different original values to a single quantized value because the range of the original values is larger than the range of the quantized values. This leads to the degradation of the accuracy of the quantized DNNs. Outliers are a main cause of the degradation of quantization resolution because they enlarge the range of original values. To solve the problem, the percentile method is often used to clip outliers. However, clipping the outliers has another problem of removing the important and strong signals in the DNNs. This paper proposes SplitQuant to keep the outliers and improve the quantization resolution at the same time. SplitQuant narrows down the range of the original values and mitigates the effect of outliers by splitting each quantizable layer into three mathematically equivalent layers and applies different scaling factors. Especially, weights and biases are clustered into lower, middle and upper clusters for optimized split. By preprocessing DNNs with SplitQuant, quantization algorithms can achieve better results. SplitQuant was applied on two BERT-Tiny models and improved the accuracy of INT2 quantization by 3.3%p and 2.1%p, achieving accuracies comparable to those of the original FP32 models.

Figures (3)

• Figure 1: SplitQuant splits each quantizable layer to three layers and combine them to improve quantization resolution while preserving functionality. (A) Original linear or convolution layer. (B) SplitQuant splits the original linear or convolution layer by clustering the weights and biases. The results from the split layers are added elementwise so that the output will be preserved. (C) Original activation layer. (D) SplitQuant splits the original activation and concatenate the results. Clustering is not possible for activation layers because the activation values can only be known in runtime. So, the original activation layer is divided into three activation layers, each with one-third of the original length. Then the results are concatenated to get the output of length.
• Figure 2: (A) Original linear layer. (B) SplitQuant runs k-means clustering on the weights (and biases if exist) to cluster them into lower, middle and upper clusters. Then three new linear layers are created from the clusters. The original linear layer and the three split layers are mathematically equivalent. The split layers have higher quantization resolutions because their ranges are smaller than the range of the original layer.
• Figure 3: (A) Original convolution layer. (B) SplitQuant runs k-means clustering on the weights (and biases if exist) to cluster them into lower, middle and upper clusters. Then three new linear layers are created from the clusters. The original convolution layer and the three split layers are mathematically equivalent. The split layers have higher quantization resolutions because their ranges are smaller than the range of the original layer.