Midtraining Bridges Pretraining and Posttraining Distributions

TL;DR

Midtraining introduces a formal intermediate phase in LM training that blends general pretraining data with higher-quality, domain-specific data to bridge the distribution gap to posttraining. Across controlled experiments with Pythia models, midtraining yields domain-specific gains—most notably in math and code—and reduces forgetting after fine-tuning compared with continued pretraining. The key finding is that the timing of midtraining data introduction matters more than the mixture weight, supporting a distributional-bridging interpretation. These results cast midtraining as an effective domain-adaptation technique, with practical guidance to introduce specialized data earlier when the target domain diverges from pretraining data, while preserving some general data for better retention; future work includes scaling to larger models and exploring curricula across multiple midtraining stages.

Abstract

Recently, many language models have been pretrained with a "midtraining" phase, in which higher quality, often instruction-formatted data, is mixed in at the end of pretraining. Despite the popularity of this practice, there is little scientific understanding of this phase of model training or why it is effective. In this work, we conduct the first systematic investigation of midtraining through controlled experiments with language models pretrained from scratch and fine-tuned on supervised finetuning datasets in different domains. We find that when compared after supervised fine-tuning, the effectiveness of midtraining is highest in the math and code domains, where midtraining can best reduce the syntactic gap between pretraining and posttraining data. In these cases, midtraining consistently outperforms continued pretraining in both in-domain validation loss as well as pretraining data forgetting after posttraining. We conduct ablations on the starting time of the midtraining phase and mixture weights of the midtraining data, using code midtraining as a case study, and find that timing has a greater impact than mixture weights, with earlier introduction of specialized data, yielding greater benefits in-domain as well as preserving general language modeling better. These findings establish midtraining as a domain adaptation technique that compared to continued pretraining yields better performance through reduced forgetting.

Figures (13)

• Figure 1: Example similarity matrix between pre/midtrain and posttraining datasets. For the complete matrix, see \ref{['appendix:dataset_similarity']}.
• Figure 2: Relationship between proximity advantage and midtraining performance improvements for pairs of midtraining and SFT datasets. Each data point represents a (midtrain, SFT) pair, where the color indicates the SFT dataset and shape represents midtrain dataset. Proximity advantage (dist(C4, SFT) - dist(midtrain, SFT)) indicates how much closer midtraining data brings the model to the target SFT dataset compared to the base pretraining data. Proximity advantage pairs near zero are greyed out for clarity but included in calculations. Relative improvement is measured against the base model pretrained on C4.
• Figure 3: Pycode ablations. (A) Timing effect (fixed 20%). (B) Mixture effect at 105B. (C) Mixture effect at 63B. Earlier start increases sensitivity to mixture weight, especially for 70M; 160M is more robust. Top: SFT after Pycode finetuning. Bottom: C4 losses (forgetting).
• Figure 4: Validation loss and C4 loss for the Starcoder-midtrained model (160M) and base pretrained model after supervised fine-tuning on the Pycode dataset, with each point on the x-axis representing the number of tokens the pretrained checkpoint was trained on.
• Figure 5: CKA analysis of model activations in the 70M model, probed with the APPS code dataset.