Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

TL;DR

Large language models require enormous compute, motivating low-precision training to reduce memory, bandwidth, and compute with minimal accuracy loss. The paper categorizes techniques by numerical format (fixed-point/integer, floating-point, customized) and covers quantization-aware training and system-level support, offering a structured taxonomy and comprehensive survey of representative methods across training components. It discusses open challenges and future directions, including non-linear and learned quantization, ultra-low precision, fine-grained scaling, and optimizer-state compression, and calls for standardized benchmarks and unified tooling. Overall, the survey highlights that a broad spectrum of precision levels—from fixed-point to ultra-low-precision floating-point and custom formats—can enable scalable, energy-efficient LLM training when paired with appropriate scaling strategies, quantization schemes, and optimizer designs.

Abstract

Large language models (LLMs) have achieved impressive performance across various domains. However, the substantial hardware resources required for their training present a significant barrier to efficiency and scalability. To mitigate this challenge, low-precision training techniques have been widely adopted, leading to notable advancements in training efficiency. Despite these gains, low-precision training involves several components$\unicode{x2013}$such as weights, activations, and gradients$\unicode{x2013}$each of which can be represented in different numerical formats. The resulting diversity has created a fragmented landscape in low-precision training research, making it difficult for researchers to gain a unified overview of the field. This survey provides a comprehensive review of existing low-precision training methods. To systematically organize these approaches, we categorize them into three primary groups based on their underlying numerical formats, which is a key factor influencing hardware compatibility, computational efficiency, and ease of reference for readers. The categories are: (1) fixed-point and integer-based methods, (2) floating-point-based methods, and (3) customized format-based methods. Additionally, we discuss quantization-aware training approaches, which share key similarities with low-precision training during forward propagation. Finally, we highlight several promising research directions to advance this field. A collection of papers discussed in this survey is provided in https://github.com/Hao840/Awesome-Low-Precision-Training.

Figures (8)

• Figure 1: Annual count of reviewed papers in this survey from 2015 to 2024, categorized by primary adopted numerical format.
• Figure 2: Overview of the survey structure and key components.
• Figure 3: Visualization of different number representation formats.
• Figure 4: Generalized model training pipeline, showcasing components that can be optimized with low-precision methods.
• Figure 5: Overview of studies on training with low-precision fixed-point and integer formats.