A probabilistic framework for dynamic quantization

TL;DR

This paper tackles the challenge of efficient input-adaptive quantization for neural networks by introducing a probabilistic framework that uses a lightweight surrogate of pre-activations to estimate quantization parameters before layer execution. The method blends static calibration with per-input adaptation through an interval-based dynamic range $I(\alpha,\beta)$ and a tunable latency parameter $\gamma$, yielding memory overhead similar to static quantization while approaching dynamic quantization performance. The authors provide analytical modeling of static vs dynamic quantization, derive estimation procedures, and validate the approach on diverse vision tasks with on-device demonstrations on microcontrollers. The results show robust performance across in-domain and out-of-domain conditions, with per-channel quantization offering particular resilience and a favorable compute-memory trade-off for resource-constrained deployments.

Abstract

We propose a probabilistic framework for dynamic quantization of neural networks that allows for a computationally efficient input-adaptive rescaling of the quantization parameters. Our framework applies a probabilistic model to the network's pre-activations through a lightweight surrogate, enabling the adaptive adjustment of the quantization parameters on a per-input basis without significant memory overhead. We validate our approach on a set of popular computer vision tasks and models, observing only a negligible loss in performance. Our method strikes the best performance and computational overhead tradeoff compared to standard quantization strategies.

Figures (5)

• Figure 1: This diagram showcases the differences between static (a), dynamic (b) and our quantization strategies (c) for a function $f(x, w)$. The quantization parameters for $w$ are always computed before model deployment. For $x$, it depends on the quantization strategy, as explained in \ref{['sec:model_q']}. The orange box indicates that the output of $f$ is already quantized using the provided parameters. Therefore, $f(x)$ is never stored as 32 variable. The blue box indicates that the parameters are computed before model deployment. The red box indicates that the parameters are computed after each evaluation of the function of interest. The green box indicates that the parameters are computed before evaluating the function $f$.
• Figure 2: This illustration showcases the effect of some of the corruptions we implemented to simulate domain shifts in the test distribution. All augmentations are shown with a severity score of three out of five.
• Figure 3: This diagram showcases the latency analysis of convolution. We vary independently the number of input and output channels. From left to right, the hyperparameters for the latency measurements were a) input shape 32x32x$C_\text{in}$, output channels 3, convolution's stride 1, b) input shape 32x32x3, convolution's stride 1, and c) input shape 32x32x3, convolution's stride 1.
• Figure 4: Impact of sampling stride, $\gamma$, on per-tensor (T) and per-channel (C) quantization.
• Figure 5: Impact of calibration dataset size, $\#\mathcal{S}$ on per-tensor (T) and per-channel (C) quantization.