Mapping Post-Training Forgetting in Language Models at Scale

TL;DR

The paper introduces a sample-wise framework to quantify forgetting and backward transfer in post-training research for language models, using $1\rightarrow0$ and $0\rightarrow1$ transitions and chance-adjusted metrics $F_{true}$ and $BT_{true}$ to separate genuine knowledge changes from random guessing. It defines ceilings $F_{max}$ and $BT_{max}$ to bound potential changes and outlines assumptions to keep the analysis robust. Across domain-continual pretraining, instruction tuning, and reasoning-based post-training, the study finds that forgetting is generally low-to-moderate, while backward transfer is often moderate-to-high in specific domains like Math and Logic, with effects scaling variably by model size and data regime. Model merging does not reliably mitigate forgetting, underscoring the need for more targeted mitigation strategies. Overall, the work provides a practical yardstick for mapping how post-training modifies pretrained knowledge at scale, guiding methods to preserve broad knowledge while acquiring new capabilities.

Abstract

Scaled post-training now drives many of the largest capability gains in language models (LMs), yet its effect on pretrained knowledge remains poorly understood. Not all forgetting is equal: Forgetting one fact (e.g., a U.S. president or an API call) does not "average out" by recalling another. Hence, we propose a sample-wise paradigm to measure what is forgotten and when backward transfer occurs. Our metric counts 1->0 transitions (correct before post-training, incorrect after) to quantify forgetting and 0->1 transitions to quantify backward transfer. Traditional task averages conflate these effects and obscure large changes. For multiple-choice benchmarks, we add chance-adjusted variants that subtract the expected contribution of random guessing from pre- and post-training accuracies. We apply this framework across post-training stages, model sizes, and data scales. Our large-scale analysis shows that: (1) Domain-continual pretraining induces moderate forgetting with low-to-moderate backward transfer; (2) RL/SFT post-training applied to base models and Instruction tuning yields moderate-to-large backward transfer on math and logic with overall low-to-moderate forgetting; (3) Applying RL/SFT to instruction-tuned models is sensitive on data scale: at small scales, both forgetting and backward transfer are small; at larger scales, effects are mixed and warrant further study with better controls; (4) Model merging does not reliably mitigate forgetting. Overall, our framework offers a practical yardstick for mapping how post-training alters pretrained knowledge at scale -- enabling progress towards generally capable AI systems.

Figures (16)

• Figure 1: Each sample is assigned to one of four quadrants by correctness before and after.
• Figure 2: Accuracy $\bar{a}$ decomposes into true knowledge $\bar{a}_{\text{true}}$ and lucky guesses $x$.
• Figure 3: Forgetting (left) and Backward Transfer (right) after domain‑continual pretraining. Forgetting is low-to-moderate and consistent across categories; backward transfer is low. Scaling model size reduces forgetting.
• Figure 4: Forgetting (left) and Backward Transfer (right) after instruction-tuning yields moderate forgetting and backward transfer categories-wise. Scaling model size reduces forgetting and backward transfer.
• Figure 5: Forgetting (left) and Backward Transfer (right) after reasoning training (SFT/RL) from base model. It generally yields minimal forgetting, except in the Culture and Knowledge categories, and has moderate to high backward‑transfer gains. Qwen2.5 Math Instruct (7B) is an exception to this trend, demonstrating forgetting across all categories.