Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ascend HiFloat8 Format for Deep Learning

Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang

TL;DR

This work introduces HiF8, an 8-bit tapered-precision floating-point format designed for deep learning that balances precision and dynamic range to support both forward and backward passes in training. It defines HiF8 with four fields (Sign, Dot, Exponent, Mantissa), a prefix-based Dot encoding, and a denormal mode to extend range, complemented by sign-magnitude exponent and implicit bits. The paper presents rounding schemes (TA and HR), plus stochastic variants, and validates HiF8 through extensive simulations on traditional neural networks and large language models, showing training and inference performance comparable to FP16 with manageable calibration costs. The findings suggest HiF8 can unify 8-bit training and inference, delivering similar accuracy with reduced data width, and point to future directions including even smaller-than-8-bit HiFloat formats for broader hardware efficiency.

Abstract

This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.

Ascend HiFloat8 Format for Deep Learning

TL;DR

This work introduces HiF8, an 8-bit tapered-precision floating-point format designed for deep learning that balances precision and dynamic range to support both forward and backward passes in training. It defines HiF8 with four fields (Sign, Dot, Exponent, Mantissa), a prefix-based Dot encoding, and a denormal mode to extend range, complemented by sign-magnitude exponent and implicit bits. The paper presents rounding schemes (TA and HR), plus stochastic variants, and validates HiF8 through extensive simulations on traditional neural networks and large language models, showing training and inference performance comparable to FP16 with manageable calibration costs. The findings suggest HiF8 can unify 8-bit training and inference, delivering similar accuracy with reduced data width, and point to future directions including even smaller-than-8-bit HiFloat formats for broader hardware efficiency.

Abstract

This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.
Paper Structure (14 sections, 4 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. HiFloat8
  3. Novel Data Format
  4. Consideration and Design
  5. Rounding Methods
  6. Rounding Half
  7. Hybrid Rounding
  8. Experiments on Traditional Neural Networks
  9. Training with Backward Loss-Scaling
  10. Inference with Per-Tensor Scaling
  11. Experiments on Large Language Models
  12. Training
  13. Inference
  14. Conclusion and Outlook

Figures (4)

  • Figure 1: Significand Precision over Exponent
  • Figure 2: Threshold and Fraction of Simplified Stochastic Rounding
  • Figure 3: Training Loss Curves for Traditional Neural Networks
  • Figure 4: Training Loss Curves for Large Language Models