Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on LLM Mid-Training

Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, Xunliang Cai

TL;DR

Mid-training is proposed as a distinct optimization stage between pre-training and post-training for LLMs, addressing the need for targeted capability gains with efficient data use. The paper formalizes a definition, proposes a five-component optimization framework (data curation, training strategies, model architecture, decay scaling laws, evaluation), and catalogues objective-driven implementations. It surveys data synthesis, long-context extension, multilingual extension, and LR scheduling as concrete methods to realize these objectives. The findings highlight data-quality balancing, RoPE-based context extension, and LR dynamics as key levers for achieving steeper gains with reduced data and compute.

Abstract

Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage that bridges pre-training and post-training. Mid-training is distinguished by its use of intermediate data and computational resources, systematically enhancing specified capabilities such as mathematics, coding, reasoning, and long-context extension, while maintaining foundational competencies. This survey provides a formal definition of mid-training for large language models (LLMs) and investigates optimization frameworks that encompass data curation, training strategies, and model architecture optimization. We analyze mainstream model implementations in the context of objective-driven interventions, illustrating how mid-training serves as a distinct and critical stage in the progressive development of LLM capabilities. By clarifying the unique contributions of mid-training, this survey offers a comprehensive taxonomy and actionable insights, supporting future research and innovation in the advancement of LLMs.

A Survey on LLM Mid-Training

TL;DR

Mid-training is proposed as a distinct optimization stage between pre-training and post-training for LLMs, addressing the need for targeted capability gains with efficient data use. The paper formalizes a definition, proposes a five-component optimization framework (data curation, training strategies, model architecture, decay scaling laws, evaluation), and catalogues objective-driven implementations. It surveys data synthesis, long-context extension, multilingual extension, and LR scheduling as concrete methods to realize these objectives. The findings highlight data-quality balancing, RoPE-based context extension, and LR dynamics as key levers for achieving steeper gains with reduced data and compute.

Abstract

Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage that bridges pre-training and post-training. Mid-training is distinguished by its use of intermediate data and computational resources, systematically enhancing specified capabilities such as mathematics, coding, reasoning, and long-context extension, while maintaining foundational competencies. This survey provides a formal definition of mid-training for large language models (LLMs) and investigates optimization frameworks that encompass data curation, training strategies, and model architecture optimization. We analyze mainstream model implementations in the context of objective-driven interventions, illustrating how mid-training serves as a distinct and critical stage in the progressive development of LLM capabilities. By clarifying the unique contributions of mid-training, this survey offers a comprehensive taxonomy and actionable insights, supporting future research and innovation in the advancement of LLMs.
Paper Structure (31 sections, 1 equation, 2 figures, 5 tables)

This paper contains 31 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Overview
  3. Data Curation
  4. Data Collection
  5. Data Synthesis
  6. Data Selection
  7. Data Decontamination
  8. Data Mixture
  9. Training Strategies
  10. Learning Rate Scheduling
  11. Other Training Settings
  12. Model Architecture Optimization
  13. Long Context Extension
  14. Position Interpolation (PI)
  15. NTK-aware Interpolation (NTK)
  16. ...and 16 more sections

Figures (2)

  • Figure 1: Mid-training has emerged as a distinct stage, separate from conventional multi-stage pre-training paradigms. Serving as a bridge between pre-training and post-training stages, mid-training enhances the specialized capabilities of pre-trained models while providing a critical transition and warm-up for subsequent post-training stages. Mid-training encompasses multiple objective-driven implementations, spanning domains such as mathematics, reasoning, coding, and more.
  • Figure 2: Pipeline of mid-training data curation.