What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs

TL;DR

The paper investigates whether loss minimization during continual pre-training (CPT) truly reflects knowledge uptake in large language systems. It introduces a distribution-matched benchmark, epoch-level probing, and a cross-model comparison to trace acquisition, retention, forgetting, and interference with unrelated skills. The findings reveal a persistent misalignment: loss monotonically improves while factual learning is unstable and non-monotonic, with high-frequency facts temporarily acquired and then forgotten and out-of-domain abilities deteriorating under continued CPT. Mechanistic circuit analyses show rapid, wholesale rewiring of knowledge pathways, explaining narrow acquisition windows and the failure to consolidate new information; a retrieval-based upper bound (RAG) demonstrates that the information is accessible at inference but not reliably internalized by optimization. The work advocates task-based stopping criteria and trajectory-level evaluation to more accurately gauge learning progress during model adaptation and to inform future CPT objectives and diagnostics.

Abstract

Continual Pre-Training (CPT) is widely used for acquiring and updating factual knowledge in LLMs. This practice treats loss as a proxy for knowledge learning, while offering no grounding into how it changes during training. We study CPT as a knowledge learning process rather than a solely optimization problem. We construct a controlled, distribution-matched benchmark of factual documents and interleave diagnostic probes directly into the CPT loop, enabling epoch-level measurement of knowledge acquisition dynamics and changes in Out-Of-Domain (OOD) general skills (e.g., math). We further analyze how CPT reshapes knowledge circuits during training. Across three instruction-tuned LLMs and multiple CPT strategies, optimization and learning systematically diverge as loss decreases monotonically while factual learning is unstable and non-monotonic. Acquired facts are rarely consolidated, learning is strongly conditioned on prior exposure, and OOD performance degrades from early epochs. Circuit analysis reveals rapid reconfiguration of knowledge pathways across epochs, providing an explanation for narrow acquisition windows and systematic forgetting. These results show that loss optimization is misaligned with learning progress in CPT and motivate evaluation of stopping criteria based on task-level learning dynamics.

Figures (15)

• Figure 1: Probing OLMo7B across continual pre-training epochs. Across training strategies, perplexity (PPL) improves monotonically, as expected; however, knowledge acquisition is unstable, and Factual Recall's gains and drops are erratically interleaved across epochs. Meanwhile, OOD Tasks Accuracy degrades monotonically through continued loss optimization.
• Figure 2: Probing LLaMA8B (top) and LLaMA1B (bottom) across continual pre-training epochs Across models, knowledge acquisition remain unstable. Factual Recall in LLaMA8B fluctuates around its pretrained baseline, while LLaMA1B exhibits transient gains followed by drops in the last epochs (7 and 9). OOD Tasks Accuracy drops immediately for all strategies except pre-training regularization.
• Figure 3: Named Entity (GPE) learning dynamics for high- and low-frequency entities via LoRA across continual pre-training epochs (OLMo). Low-frequency entities show minimal learning, while high-frequency entities exhibit non-monotonicity across epochs, alternating between learning ($\text{recall} \geq 60\%$) and forgetting ($\text{recall} < 60\%$). Results of other models and entity categories are shown in §Figures \ref{['fig:olmo_lora_freq_heat_person']}--\ref{['fig:llama3-8b_lora_freq_heat_person']}).
• Figure 4: Extended continual pre-training of LLaMA1B on the main entity documents for 100 epochs via LoRA.PPL decreases monotonically and plateaus early, while Factual Recall peaks at epoch 3–5 and then collapses to a low, oscillatory steady state. OOD Tasks Accuracy drops sharply within the first 5–10 epochs and does not recover.
• Figure 5: Jaccard similarity between the base model knowledge circuit $C_0$ and the circuits extracted at each epoch of CPT $C_t$ (LLaMA1B). Circuit overlap drops sharply after the first epoch, with the majority of original edges being replaced immediately.