ECQ$^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs

TL;DR

ECQ$^{\text{x}}$ addresses the memory and compute bottlenecks of DNNs on edge hardware by fusing explainability with entropy-aware quantization. It extends entropy-constrained quantization (ECQ) by incorporating per-weight relevances from Layer-wise Relevance Propagation (LRP) into the assignment of weights to quantization clusters, enabling selective preservation of highly relevant weights while aggressively sparsifying others. The approach leverages a modified objective $A^{(l)} = \underset{c}{\text{argmin}} \, d(\mathbf{W}^{(l)}, w_c^{(l)}) - \lambda^{(l)} \log_2(P_c^{(l)})$ with a relevance-weighted cost and a regrowth mechanism governed by parameters $\rho$ and $p$, trained with quantization-aware training and STE. Empirical results across MLP_GSC, VGG16, and ResNet18 on GSC, CIFAR-10, and Pascal VOC show substantial sparsity and up to $103\times$ compression without compromising accuracy (and sometimes even improving it), highlighting the practical impact for edge AI and hardware-aware deployment.

Abstract

The remarkable success of deep neural networks (DNNs) in various applications is accompanied by a significant increase in network parameters and arithmetic operations. Such increases in memory and computational demands make deep learning prohibitive for resource-constrained hardware platforms such as mobile devices. Recent efforts aim to reduce these overheads, while preserving model performance as much as possible, and include parameter reduction techniques, parameter quantization, and lossless compression techniques. In this chapter, we develop and describe a novel quantization paradigm for DNNs: Our method leverages concepts of explainable AI (XAI) and concepts of information theory: Instead of assigning weight values based on their distances to the quantization clusters, the assignment function additionally considers weight relevances obtained from Layer-wise Relevance Propagation (LRP) and the information content of the clusters (entropy optimization). The ultimate goal is to preserve the most relevant weights in quantization clusters of highest information content. Experimental results show that this novel Entropy-Constrained and XAI-adjusted Quantization (ECQ$^{\text{x}}$) method generates ultra low-precision (2-5 bit) and simultaneously sparse neural networks while maintaining or even improving model performance. Due to reduced parameter precision and high number of zero-elements, the rendered networks are highly compressible in terms of file size, up to $103\times$ compared to the full-precision unquantized DNN model. Our approach was evaluated on different types of models and datasets (including Google Speech Commands, CIFAR-10 and Pascal VOC) and compared with previous work.

Figures (10)

• Figure 1: Difference in sensitivity between activation and weight quantization of the EfficientNet-B0 model pre-trained on ImageNet. As a quantization scheme uniform quantization without re-training was used. Activations are more sensitive to quantization since model performance drops significantly faster. Going below 8 bit is challenging and often requires (quantization-aware) re-training of the model to compensate for the quantization error. Data originates from fanta4.
• Figure 2: Quantizing a neural network’s layer weights (binned weight distribution shown as green bars) to 7 discrete cluster centers (centroids). The centroids (black bars) were generated by k-means clustering and the height of each bar represents the number of layer weights which are assigned to the respective centroid.
• Figure 3: LRP can be utilized to calculate relevance scores for weight parameters $W$, which contribute to the activation of output neurons $z_j$ during the forward pass in interaction with data-dependent inputs $a_i$. In the backward pass, relevance messages $R_{i\leftarrow j}$ can be aggregated at neurons / input activations $a_i$, but also at weights $W$.
• Figure 4: Weight relevance $R_{w_{ij}}$ vs. weight value $w_{ij}$ for the input layer (left) and output layer (right) of the full-precision MLP$\_$GSC model (introduced in Section \ref{['subsec:exp_setup']}). The black histograms to the top and right of each panel display the distributions of weights (top) and relevances (right). The blue histograms further show the amount of relevance (blue) of each weight histogram bin. All relevances are collected over the validation set with equally weighted samples (i.e., by choosing $R_n = 1$). The value $c$ measures the Pearsson correlation coefficient between weights and relevances.
• Figure 5: Exemplary ECQ$^{\text{x}}$ weight update. For simplicity, 3 centroids are used (i.e., symmetric 2 bit case). The process involves the following steps: 1) Derive gradients and LRP relevances from forward-backward pass. 2) LRP relevance scaling. 3) Gradients scaling. 4) Gradient attachment to full precision background model. 5) Background model update and nearest-neighbor clustering. 6) Computing of the assignment cost for each weight using the $\lambda$-scaled information content of clusters and the $\rho$-scaled relevances. Assign each weight by minimizing the cost. 7) Choosing an appropriate candidate (of various $\lambda$ and $\rho$ settings).