Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors

TL;DR

The paper addresses the challenge of deploying ultra-low-latency deep learning on resource-constrained FPGA hardware for particle detectors by proposing automatic heterogeneous quantization at the per-layer and per-parameter level. It introduces QKeras for quantization-aware training, AutoQKeras for hardware-aware automatic quantization with an energy/bit-size objective and a forgiving factor, and QTools for energy estimation, all integrated with hls4ml to produce on-chip firmware. The approach achieves substantial resource reductions (up to ~50×) with minimal accuracy loss, and demonstrates nanosecond-scale inference suitable for LHC trigger systems, highlighting practical impact for edge AI in physics experiments. The work also outlines paths for broader applicability and future energy modeling enhancements, including cross-hardware integration and more precise device-specific estimates.

Abstract

Although the quest for more accurate solutions is pushing deep learning research towards larger and more complex algorithms, edge devices demand efficient inference and therefore reduction in model size, latency and energy consumption. One technique to limit model size is quantization, which implies using fewer bits to represent weights and biases. Such an approach usually results in a decline in performance. Here, we introduce a method for designing optimally heterogeneously quantized versions of deep neural network models for minimum-energy, high-accuracy, nanosecond inference and fully automated deployment on chip. With a per-layer, per-parameter type automatic quantization procedure, sampling from a wide range of quantizers, model energy consumption and size are minimized while high accuracy is maintained. This is crucial for the event selection procedure in proton-proton collisions at the CERN Large Hadron Collider, where resources are strictly limited and a latency of ${\mathcal O}(1)~μ$s is required. Nanosecond inference and a resource consumption reduced by a factor of 50 when implemented on field-programmable gate array hardware are achieved.

Figures (7)

• Figure I: An ultra-compressed deep neural network for particle identification on a Xilinx FPGA.
• Figure II: Model architecture for the fully-connected NN architecture under study. The numbers in brackets are the precisions used for each layer, quoted as $\langle B,I \rangle$, where $B$ is the precision in bits and $I$ the number of integer bits. When different precision is used for weights and biases, the quantization is listed as w and b, respectively. These have been obtained using the per-layer, per-parameter type automatic quantization procedure described in Section \ref{['sec:autoq']}.
• Figure III: The quantized_relu function as implemented in QKeras for a 2- (purple), 3-bit (green and blue) and 6-bit (yellow) precision and for 0 or 1 integer bits. The unquantized ReLU function is shown for comparison (orange).
• Figure IV: ROC curves of false positive rate (FPR) versus true positive rate (TPR) for the Baseline Full (BF), quantized 6-bit (Q6), AutoQKeras Energy Optimized (QE) and AutoQKeras Bits Optimized (QB) models.
• Figure V: The full workflow starting from a baseline TensorFlow Keras Model, which is then converted into an optimally quantized equivalent through QKeras and AutoQKeras. This model is then translated into highly parallel firmware with hls4ml.