Histogram-Equalized Quantization for logic-gated Residual Neural Networks

TL;DR

This work tackles accuracy losses in quantized networks by introducing Histogram-Equalized Quantization (HEQ), an adaptive scheme that updates the single step size $s$ during training to balance $n$-level quantization within the range $[-1,1]$. HEQ leverages $n$-quantiles of proxy weights to align quantization thresholds with the weight distribution, effectively equalizing the histogram of quantized values. Empirically, HEQ achieves state-of-the-art results on CIFAR-10 and improves STL-10 performance when coupled with logic-gated skip connections that replace costly full-precision adds with simple hardware-friendly gates. The approach generalizes across architectures, providing hardware-efficient quantization that maintains or improves accuracy while reducing design complexity for edge deployments.

Abstract

Adjusting the quantization according to the data or to the model loss seems mandatory to enable a high accuracy in the context of quantized neural networks. This work presents Histogram-Equalized Quantization (HEQ), an adaptive framework for linear symmetric quantization. HEQ automatically adapts the quantization thresholds using a unique step size optimization. We empirically show that HEQ achieves state-of-the-art performances on CIFAR-10. Experiments on the STL-10 dataset even show that HEQ enables a proper training of our proposed logic-gated (OR, MUX) residual networks with a higher accuracy at a lower hardware complexity than previous work.

Figures (5)

• Figure 1: Weight distributions of 2 layers (in 2 columns) after training of: full-precision model ($1^{st}$ line), existing ternary-weight and quinary-weight model ($2^{nd}$ line), and our proposed HEQ method ($3^{rd}$ line) along with quantization thresholds.
• Figure 2: Symmetric linear quantization with histogram bin equalization when $n$-quantiles ($q_{-i}, q_i$) are symmetrical and coincide with the quantized thresholds.
• Figure 3: Comparison of the ternary-weight distribution using TWN and our HEQ method.
• Figure 4: Evolution of the step size $s$ during training.
• Figure 5: Models with the plain block (11-hidden layer VGGVGGNet-variant), OR-gated block and MUX-OR gated block.