Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training

TL;DR

This work tackles the high training cost of deep networks by introducing Quantization-Aware Approximate Log Number System (QAA-LNS), a bitwidth-specific arithmetic for logging-domain representations. It designs a hardware-friendly, piecewise-linear approximation for the log-add operation, optimized per bitwidth via simulated annealing against a quantization-aware loss $L(\underline{x},\underline{y})$. End-to-end training with QAA-LNS on CNNs demonstrates that $12$-bit configurations can approach floating-point accuracy in some cases, while more aggressive bitwidths may require higher precision or diverge, revealing a trade-off between compression and stability. A hardware study reports substantial area and energy savings for LNS MACs relative to linear fixed-point and FP implementations, highlighting practical benefits for accelerator design. Overall, the work provides a viable path toward scalable low-bitwidth training through bitwidth-aware logarithmic arithmetic and motivates further hardware-aware optimization.

Abstract

While advancements in quantization have significantly reduced the computational costs of inference in deep learning, training still predominantly relies on complex floating-point arithmetic. Low-precision fixed-point training presents a compelling alternative. This work introduces a novel enhancement in low-precision logarithmic fixed-point training, geared towards future hardware accelerator designs. We propose incorporating bitwidth in the design of approximations to arithmetic operations. To this end, we introduce a new hardware-friendly, piece-wise linear approximation for logarithmic addition. Using simulated annealing, we optimize this approximation at different precision levels. A C++ bit-true simulation demonstrates training of VGG-11 and VGG-16 models on CIFAR-100 and TinyImageNet, respectively, using 12-bit integer arithmetic with minimal accuracy degradation compared to 32-bit floating-point training. Our hardware study reveals up to 32.5% reduction in area and 53.5% reduction in energy consumption for the proposed LNS multiply-accumulate units compared to that of linear fixed-point equivalents.

Figures (2)

• Figure 1: Plot of the $\Delta_{+}(d)$ and $\Delta_{-}(d)$ functions (a) with an example of a bit-shift approximation and our proposed piece-wise linear approximation with power-of-two slopes (b). Notice how both $\Delta_{+}(d)$ and $\Delta_{-}(d)$ functions converge to zero within a small input range. This allows us to design an approximation over a small range without increasing quantization error. Also notice in (b) how the piece-wise linear segments provide a more accurate approximation to the curves.
• Figure 2: Validation accuracy curves with ResNet-18 on TinyImageNet for multiple experiments. To reach the floating-point level, at least $14$ 'arithmetic' bits are needed. $12$-bit LNS suffers from severe degradation in accuracy. 'Not QA' refers to using an approximation that is not 'quantization-aware'. '$11$-bit QA' refers to using a 'quantization-aware' approximation that was optimized for $11$-bit LNS.

Theorems & Definitions (3)

• Definition 3.1: LNS Fixed-Point Representation
• Definition 3.2: LNS Multiplication
• Definition 3.3: LNS Addition