Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

TL;DR

This work introduces MS-EDEN, a microscale unbiased quantization technique for NVFP4 that reduces quantization error relative to stochastic rounding, enabling a fully NVFP4 pre-training pipeline named Quartet II. Quartet II combines an enhanced forward pass with Four-Over-Six scale selection and an MS-EDEN-based unbiased backward pass, yielding consistently better gradient estimates across GEMMs and improved end-to-end LLM pre-training results up to 1.9B parameters on 38B tokens. The approach is supported by hardware-focused CUDA kernels on NVIDIA Blackwell GPUs, achieving up to 4.2x BF16 speedups in certain configurations. Across Llama-like models and Nanochat, Quartet II demonstrates stable training, reduced loss gaps to BF16, and practical throughput gains, illustrating the viability of fully quantized NVFP4 training for large-scale LLMs.

Abstract

The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs. Yet, existing quantized training methods still sacrifice some of the representation capacity of this format in favor of more accurate unbiased quantized gradient estimation by stochastic rounding (SR), losing noticeable accuracy relative to standard FP16 and FP8 training. In this paper, improve the state of the art for quantized training in NVFP4 via a novel unbiased quantization routine for micro-scaled formats, called MS-EDEN, that has more than 2x lower quantization error than SR. We integrate it into a novel fully-NVFP4 quantization scheme for linear layers, called Quartet II. We show analytically that Quartet II achieves consistently better gradient estimation across all major matrix multiplications, both on the forward and on the backward passes. In addition, our proposal synergizes well with recent training improvements aimed specifically at NVFP4. We further validate Quartet II on end-to-end LLM training with up to 1.9B parameters on 38B tokens. We provide kernels for execution on NVIDIA Blackwell GPUs with up to 4.2x speedup over BF16. Our code is available at https://github.com/IST-DASLab/Quartet-II .

Figures (9)

• Figure 1: Impact of selective NVFP4 backward pass quantization on C4 Validation Loss relative to BF16 pre-training for $N$-parameter Llama-2-like LLMs with $D/N$ tokens-per-parameter. Axis captions indicate which tensors of the two backward pass GEMMs are quantized.
• Figure 2: NVFP4 Forward Pass C4 Validation Loss Gaps relative to BF16 pre-training for $N$-parameter Llama-2-like LLMs with $D/N$ tokens-per-parameter. "16x16gs" and "1x16gs" indicate whether square block quantization was used or not and "+4/6" indicates whether Four Over Six cook2025sixaccuratenvfp4quantization was used.
• Figure 3: Quartet II fully-NVFP4 linear layer computation scheme.
• Figure 4: Fully-NVFP4 (forward pass and backward pass) C4 Validation Loss Gaps relative to BF16 pre-training for $N$-parameter Llama-2-like LLMs with $D/N$ tokens-per-parameter for Quartet II and baselines.
• Figure 5: Validation loss curves for Nanochat pre-training. Plot show relative increase in bits-per-byte (BPB) w.r.t. BF16 pre-training. Loss spikes are observed for both BF16 and QAT around 6T tokens but training stabilizes later.

Theorems & Definitions (1)

• Corollary 3.1