An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

TL;DR

This study systematically probes how noisy data affects LLM pretraining stability by injecting uniform random noise into a clean corpus across models from $480M$ to $5.2B$ parameters. It shows that noise can cause loss divergence, with probability depending on noise type, magnitude $\alpha$, and model depth, and introduces diagnostics to distinguish noisy-data divergences from high learning-rate failures. The authors Compare dense and MoE architectures, find similar sensitivity to noise, and demonstrate interventions such as data cleaning and $\text{QK-layernorm}$ that stabilize training. The work provides practical guidance for data curation and model design to enhance robustness in large-scale pretraining.

Abstract

Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulated web content or randomness inherent in data. Although LLM pretrainers often speculate that such noise contributes to instabilities in large-scale LLM pretraining and, in the worst cases, loss divergence, this phenomenon remains poorly understood.In this work, we present a systematic empirical study of whether noisy data causes LLM pretraining divergences and how it does so. By injecting controlled synthetic uniformly random noise into otherwise clean datasets, we analyze training dynamics across model sizes ranging from 480M to 5.2B parameters. We show that noisy data indeed induces training loss divergence, and that the probability of divergence depends strongly on the noise type, amount of noise, and model scale. We further find that noise-induced divergences exhibit activation patterns distinct from those caused by high learning rates, and we provide diagnostics that differentiate these two failure modes. Together, these results provide a large-scale, controlled characterization of how noisy data affects loss divergence in LLM pretraining.

Figures (29)

• Figure 1: Examples of four pretraining runs using the same 1.3B model architecture and 15% noisy data, differing only in random seed. The left two runs illustrate cases where training diverges, while the right two show stable behavior. We focus on divergences rather than fast spikes, so the third run is categorized as stable because its loss spikes quickly recover and the loss continues to decrease.
• Figure 2: Effect of noise vocabulary size on training stability for 540M dense models with noise ratio ${\alpha = 55\%}$. We observe that reducing the size of the noise vocabulary significantly increases the probability of divergence. In all plots throughout the paper, we use shaded regions to denote the standard error.
• Figure 3: Effect of noise vocabulary content on training stability. Fixing the noise vocabulary size to $|V_n|{=}5$, we plot the divergence rate against the average frequency of the selected noise tokens in the clean corpus. We observe that the content of the noise vocabulary has little effect on loss divergence, with a near zero Pearson correlation.
• Figure 4: Comparison of inserting versus overwriting noise for a fixed noise vocabulary of five tokens. Inserting noise results in a higher probability of loss divergence than overwriting noise.
• Figure 5: Scaling model size by jointly scaling the depth and width of the model. We observe that larger models are more sensitive to noise, and a higher noise ratio leads to more divergences.