AMED: Automatic Mixed-Precision Quantization for Edge Devices

TL;DR

AMED tackles edge-device quantization by reconceptualizing mixed-precision allocation as a hardware-guided, stochastic process. By modeling per-layer bitwidths as a multivariate Markov chain and optimizing a non-differentiable objective that combines accuracy and latency, AMED directly leverages simulator signals to find Pareto-efficient bit allocations. The approach integrates a hardware-aware quantizer and a SCALE-Sim-based accelerator simulator within a Metropolis–Hastings search, followed by lightweight fine-tuning. Empirical results on ImageNet show AMED achieving superior accuracy-latency trade-offs across ResNet-18, ResNet-50, and MobileNetV2 compared to state-of-the-art quantization methods, with per-hardware bit allocations adapting to memory and compute bottlenecks. This demonstrates practical impact for deploying high-accuracy networks on diverse edge devices while reducing latency and memory footprints.

Abstract

Quantized neural networks are well known for reducing the latency, power consumption, and model size without significant harm to the performance. This makes them highly appropriate for systems with limited resources and low power capacity. Mixed-precision quantization offers better utilization of customized hardware that supports arithmetic operations at different bitwidths. Quantization methods either aim to minimize the compression loss given a desired reduction or optimize a dependent variable for a specified property of the model (such as FLOPs or model size); both make the performance inefficient when deployed on specific hardware, but more importantly, quantization methods assume that the loss manifold holds a global minimum for a quantized model that copes with the global minimum of the full precision counterpart. Challenging this assumption, we argue that the optimal minimum changes as the precision changes, and thus, it is better to look at quantization as a random process, placing the foundation for a different approach to quantize neural networks, which, during the training procedure, quantizes the model to a different precision, looks at the bit allocation as a Markov Decision Process, and then, finds an optimal bitwidth allocation for measuring specified behaviors on a specific device via direct signals from the particular hardware architecture. By doing so, we avoid the basic assumption that the loss behaves the same way for a quantized model. Automatic Mixed-Precision Quantization for Edge Devices (dubbed AMED) demonstrates its superiority over current state-of-the-art schemes in terms of the trade-off between neural network accuracy and hardware efficiency, backed by a comprehensive evaluation.

Figures (10)

• Figure S1: ResNet-50 quantized models on a latency--accuracy plane. Circles are mixed-precision quantization, and triangles are uniform quantization. Our models achieve a better Pareto curve of dominant solutions in the two-dimensional plane for ultra-low precision.
• Figure S2: A diagram of DNN architecture activation maps.
• Figure S3: Quantization of the $i$-th layer of the network. $W_i$ indicates the weights, $X_i$ the input activations, and $\Bar{W_i}, \Bar{X_i}$ the quantized versions, respectively. $S_{W_i},S_{X_i}$ are the learnable parameters of the quantization. (scale). In Blue are dynamically changed tensors, while in orange the parameters. Weights and activations are quantized with respect to the scaling factor (with rounding and clamping as described in \ref{['method: quantizer']}. The quantized versions are multiplied in an integer matrix multiplication accelerator and produce a quantized vector $\Bar{Z_i}$. With respect to the scaling factors, we can dequantize them into $Z_i$, which can yield a prediction in FP, or quantize them again in a different precision in the next layer.
• Figure S4: An illustration of Algorithm \ref{['alg:algo']}. $\hat{\mathcal{Q}}$ Table represents bit-allocation vector $\bm{\mathrm{A}}$
• Figure S5: Quantization bit allocation of MobileNetV2 following our method using the simulator. The top figure is the SCALE-Sim setup; the middle is the Eyeriss setup; the bottom is SCALE-Sim with low memory. Depthwise convolutions have a higher feature map and, thus, higher memory footprint, and we can see that Algorithm \ref{['alg:algo']} allocates fewer bits when the system memory is low, i.e., the model is memory-bounded. Models with higher memory allocate the bits differently due to the locality of the boundary (memory or computational) by the layer. This figure does not include the first and last layers, which we quantize to 8 bits.