Quantized and Interpretable Learning Scheme for Deep Neural Networks in Classification Task

TL;DR

The paper tackles deploying deep neural networks in resource-limited settings while preserving interpretability. It combines Saliency-Guided Training (SGT), which masks low-gradient features and enforces output consistency via KL divergence, with Parameterized Clipping Activation (PACT) quantization to achieve $k$-bit activations and weights. The approach is evaluated on MNIST and CIFAR-10 using ResNet-20, showing near-full-precision accuracy with significant gains in efficiency and clearer saliency maps; dynamic optimization of the clipping parameter $\\alpha$ further balances accuracy and compression. Overall, the work demonstrates that interpretable, resource-efficient DNNs can be realized, enabling deployable transparent AI on edge devices.

Abstract

Deep learning techniques have proven highly effective in image classification, but their deployment in resourceconstrained environments remains challenging due to high computational demands. Furthermore, their interpretability is of high importance which demands even more available resources. In this work, we introduce an approach that combines saliency-guided training with quantization techniques to create an interpretable and resource-efficient model without compromising accuracy. We utilize Parameterized Clipping Activation (PACT) to perform quantization-aware training, specifically targeting activations and weights to optimize precision while minimizing resource usage. Concurrently, saliency-guided training is employed to enhance interpretability by iteratively masking features with low gradient values, leading to more focused and meaningful saliency maps. This training procedure helps in mitigating noisy gradients and yields models that provide clearer, more interpretable insights into their decision-making processes. To evaluate the impact of our approach, we conduct experiments using famous Convolutional Neural Networks (CNN) architecture on the MNIST and CIFAR-10 benchmark datasets as two popular datasets. We compare the saliency maps generated by standard and quantized models to assess the influence of quantization on both interpretability and classification accuracy. Our results demonstrate that the combined use of saliency-guided training and PACT-based quantization not only maintains classification performance but also produces models that are significantly more efficient and interpretable, making them suitable for deployment in resource-limited settings.

Figures (3)

• Figure 1: $\alpha$ Optimization for MNIST
• Figure 2: $\alpha$ Optimization for CIFAR10
• Figure 3: Accuracy drop comparison across different models on MNIST dataset with varying levels of masking percentages. It is shown that our approach's accuracy drops more as the masking percentage increases. This is an illustration that the model has learned more salient features during the training.