Do we really have to filter out random noise in pre-training data for language models?

TL;DR

This work systematically investigates random noise in web-scale pretraining data for language models, revealing that random noise modestly increases the next-token prediction loss even for large models. It develops a theoretical framework based on a contamination model and demonstrates that the global optimum is robust to moderate noise under realistic mixing proportions, with extensions to multilingual and multimodal settings. To mitigate potential downstream degradation, the authors propose Local Gradient Matching (LGM), a plug-and-play loss that aligns the gradients of clean and perturbed downstream features, providing theoretical bounds and broad empirical gains across NLP and vision tasks under black-box fine-tuning. The findings suggest that while random noise is not catastrophic for pretraining, careful downstream denoising can further enhance generalization, supporting a data-centric approach to AI development.

Abstract

Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content. In contrast to previous works that focus on low quality or synthetic data, our study \textbf{provides the first systematic investigation of such random noise through a cohesive ``What-Why-How'' framework.} Surprisingly, we observed that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B. We provide a theoretical justification for this phenomenon, which also elucidates the success of multilingual models and can be applied to multimodal models. On the other hand, experiments show that the model's performance in downstream tasks is not based solely on the NTP loss, which means that random noise may result in degraded downstream performance. To address the potential adverse effects, we introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters. Additional experiments on 8 language and 14 vision benchmarks further validate its effectiveness.

Figures (9)

• Figure 1: Overview of the study and methodology. (a) The common scenario in which a GPT model, pre-trained on filtered data $P^c$, demonstrates robust performance. (b) When the pre-training dataset is contaminated with random noise $P^n$, the resultant language model may exhibit unpredictable behavior. (c) Our approach follows nml and focuses on the effective fine-tuning of black-box noisy models for downstream tasks $P^d$.
• Figure 2: Next-token prediction loss on the clean OpenWebText validation set for GPT-2 124M models pre-trained on synthetic OpenWebText datasets with varying levels of random noise. (a) Trend of NTP loss as training proceeds. (b) Difference in NTP loss between the noisy and clean models after the same number of training iterations. (c) Difference in loss values after undergoing the same number of training iterations on clean OpenWebText data.
• Figure 3: Validation experiments with the models trained with 5% random noise. (a) Loss trends on the OpenWebText validation set for GPT-2 2.7B model. (b) Comparison of the 124M training set loss between 5% random noise and Gaussian noise. (c) Loss difference on the validation set for 124M models trained on datasets with 5% random noise and 5% Gaussian noise, respectively.
• Figure 4: Overview of the proposed Local Gradient Mathcing scheme.
• Figure 5: Visualization of input sensitivity for models trained with (a) no (b) $L^2$ (c) $\mathcal{L}_{gm}$ regularization. We randomly select a sample and introduce perturbations on a two-dimensional hyperplane, where different colors represent different labels, and green indicates the correct label.

Theorems & Definitions (10)

• Proposition 1
• Proposition 2
• Lemma 1
• proof
• proof
• Definition 1: $\beta$-smooth unlabeled
• Definition 2: $\rho$-input flatness
• Lemma 2
• proof
• proof