Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning

TL;DR

This work tackles the opaque relationship between pretraining data and large language model capabilities by introducing GRACE, a gradient ascent–based machine unlearning framework augmented with retraining to measure data influence while preserving non-target knowledge. By analyzing 48 target corpora across 31 benchmarks and using Llama-2-7B, the authors identify high-impact data (notably Books, Shell, and GitHub) and reveal how corpora interact as correlated, complementary, or orthogonal, shaping both individual abilities and joint performance. They further show that corpus organization, data correlations, and training order materially affect pretraining efficiency and capability emergence, offering concrete guidelines for data proportioning, scheduling, and evaluation. The findings provide a data-driven foundation for more efficient pretraining and point to directions for extending machine unlearning analyses to broader model families and training stages.

Abstract

Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of ``high-impact data'' such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs.

Figures (11)

• Figure 1: The impact of Unlearning different types of corpus on different abilities of the llama2-7B model.
• Figure 2: Comparative Analysis of Model Unlearning Effects between the $target_{math}$ and $non\text{-}target_{math}$.
• Figure 3: The overall framework of the experiment.
• Figure 4: The Top and Bottom 5 datasets that have the most and the least impact on each type of model capability.
• Figure 5: A correlation matrix based on the model's performance across 19 capabilities after experiencing data Unlearning. Among them, "Algorithm" is the average value of all Algorithm.