How Do Large Language Models Acquire Factual Knowledge During Pretraining?

TL;DR

The observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step, however, this increase is diluted by subsequent forgetting.

Abstract

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus.

Figures (29)

• Figure 1: An illustration of the change of log probability of the target span of a probe ($\Delta \ell(q)$) measuring the memorization of factual knowledge on a short-term scale. At step 0 (marked as a dotted line), the model is trained with the injected knowledge which contains the factual knowledge evaluated by the probe $q$. The local acquisition maxima (marked as a red line) is the timestep where the log probability reaches its maximum within the window (shaded area), defined by $t_w$. The measurement of effectivity and retainability at $t=30$ is visualized, where retainability is obtained by measuring the fraction of the purple line compared to the gray line.
• Figure 2: Change in the average log probability of target spans of the probes plotted against training steps during the continuation of pretraining OLMo-7B mid checkpoint (trained on 500B tokens) with injecting the knowledge in the Fictional Knowledge dataset. Results are shown for duplicate (Top), paraphrase (Center), and once (Bottom) injection scenarios. Note the immediate and distinctive increase of log probability after the model is updated with the injected knowledge, marked by dotted vertical lines.
• Figure 3: Effectivity averaged across various probes and each time of injection, measured for different injection scenarios, and acquisition depths. Note that the effectivity does not improve as the model is trained with more tokens (Left), whereas there is a clear improvement as the model size scales (Right).
• Figure 4: Average retainability against training steps past the local acquisition maxima, measured with OLMo-7B mid checkpoint. The x-axes are in log scale. Left: duplication. Right: paraphrase.
• Figure 5: Comparison of the forgetting dynamics of pretraining (Left) and training with reduced batch size (Right), measured with OLMo-7B mid checkpoint. Note that the x-axis represents the number of training tokens instead of training steps, which has a shifting effect on the data plotted in Figure \ref{['fig:forgetting_pretrain']}.

Theorems & Definitions (3)

• Definition 1
• Definition 2
• Definition 3