8-bit Numerical Formats for Deep Neural Networks

TL;DR

The paper systematically evaluates 8-bit floating-point formats for mixed-precision training, showing that careful selection of formats (notably 1.4.3 for activations/weights and 1.5.2 for gradients) and appropriate exponent biases can preserve float-32 accuracy on both image and language tasks. It demonstrates that loss scaling, including automatic strategies like Backoff and LogMax, further stabilizes training in low precision. Across CNNs (e.g., ResNet and EfficientNet) and NLP models (Transformer-based systems and BERT variants), 8-bit FP training achieves parity with higher-precision baselines while enabling significant reductions in memory bandwidth and energy consumption. These findings support using 8-bit floating-point mixed precision as a practical route to faster, more power-efficient training without sacrificing performance.

Abstract

Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.

Figures (26)

• Figure 1: Illustration of the quantization procedure (a) for fully connected and convolutional layers and (b) for matrix multiplications in attention layers.
• Figure 2: ResNet-32 CIFAR-100 test performance of two different 8-bit floating-point formats compared to the scaled integer format (1.0.7) for representation of the activations without quantization of inputs to the first layer (a, b, c), and of the weights (d, e, f). Test accuracy mean $\pm$ standard deviation over ten independent runs.
• Figure 3: ResNet-32 CIFAR-100 test performance of two different 8-bit floating-point formats compared to the scaled integer format (1.0.7) for representation of the gradients with respect to activations (a, b, c), and of the gradients with respect to weights (d, e, f). Test accuracy mean $\pm$ standard deviation over ten independent runs.
• Figure 4: ResNet-32 CIFAR-100 test performance relative to reference float-32 performance. Weights/activations quantization format 1.4.3 with biases 10, 14 respectively. Loss gradients with respect to weights/activations quantization format 1.5.2 with biases indicated on the x and y axes.
• Figure 5: Transformer WMT14 English-German translation performance of different 8-bit floating-point formats for representation of the activations (a, b), and the weights (c, d). BLEU mean $\pm$ standard deviation over five independent runs.