Dissecting Outlier Dynamics in LLM NVFP4 Pretraining

TL;DR

This work analyzes outlier dynamics during NVFP4 pretraining, showing that Linear Attention reduces global heavy tails compared to Softmax Attention but still exhibits block-level spikes and post-QK sensitivity. It identifies Softmax, gating, SwiGLU, and RMSNorm as primary outlier sources and documents the evolution from drifting spikes to persistent hot channels. The authors introduce Hot-Channel Patch and the CHON recipe to online compensate hot channels and protect post-QK operations, achieving substantial reductions in the BF16 loss gap and preserving downstream accuracy. The results demonstrate that CHON provides a robust, hardware-friendly approach for NVFP4 pretraining, with quantified efficiency gains and transferability to downstream tasks.

Abstract

Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve temporally. We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with "post-QK" operations exhibiting higher sensitivity to quantization. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B model trained for 60B tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.

Figures (34)

• Figure 1: Activation kurtosis of GLA vs. Qwen2.5. Lower kurtosis values in GLA reflect reduced heavy-tailedness and fewer systematic outliers across both attention and MLP layers.
• Figure 2: Architectural overview and sensitivity analysis. Detailed components of (a) residual blocks, (b) Softmax Attention, (c) Gated Linear Attention, and (d) gated-FFN. Color coding identifies outlier sources (green) and sensitive "post-QK" operations (red), providing a basis for targeted precision allocation in the CHON recipe.
• Figure 3: Temporal evolution of activation outliers in Up and Output projections. Red markers illustrate a clear transition from transient, drifting spikes (Step 400) to persistent, spatially fixed "hot channels" (Step 15k).
• Figure 4: Comparison of block-level kurtosis for Softmax and Linear Attention. Distribution of min, avg, and max kurtosis across components. Localized "heavy tails" persist at the block level even when per-tensor kurtosis remains moderate.
• Figure 5: Evolution of per-tensor kurtosis. GLA-1.3B (blue) demonstrates superior weight distribution stability compared to Qwen3-1.7B (red), whereas activation kurtosis exhibits similar stochastic volatility across both architectures.

Theorems & Definitions (26)

• Proposition 4.1: Error decomposition of a quantized product
• Definition 1.1: Hot-channel index set
• Definition 1.2: Selection operator and residual patch on $\mathcal{I}$
• Lemma 1.3: No compensation (baseline)
• proof
• Lemma 1.4: First-order activation compensation
• proof
• Lemma 1.5: Second-order weight–activation compensation
• proof
• Definition 1.6: Mean-squared error (MSE)