Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Merging in Pre-training of Large Language Models

Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, Jin Ma, Xunhao Lai, Deyi Liu, Yao Luo, Xingyan Bin, Hongbin Ren, Mingji Han, Wenhao Hao, Bairen Yi, LingJun Liu, Bole Ma, Xiaoying Jia, Xun Zhou, Siyuan Qiao, Liang Xiang, Yonghui Wu

TL;DR

The work introduces Pre-trained Model Averaging (PMA) to merge checkpoints during LLM pre-training, covering dense and MoE models from millions to >100B parameters. It demonstrates that merging during stable, constant-learning-rate phases yields consistent performance gains and can emulate annealing, reducing training costs and speeding validation. Through extensive ablations, it analyzes merging strategies, interval and model-count choices, and the downstream impact, while introducing PMA-init to stabilize subsequent training and recover from loss spikes. The study provides practical guidelines for pre-training merging and offers a mechanistic view of why merging works via a Taylor-series-like exploration of the loss landscape.

Abstract

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging.

Model Merging in Pre-training of Large Language Models

TL;DR

The work introduces Pre-trained Model Averaging (PMA) to merge checkpoints during LLM pre-training, covering dense and MoE models from millions to >100B parameters. It demonstrates that merging during stable, constant-learning-rate phases yields consistent performance gains and can emulate annealing, reducing training costs and speeding validation. Through extensive ablations, it analyzes merging strategies, interval and model-count choices, and the downstream impact, while introducing PMA-init to stabilize subsequent training and recover from loss spikes. The study provides practical guidelines for pre-training merging and offers a mechanistic view of why merging works via a Taylor-series-like exploration of the loss landscape.

Abstract

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging.
Paper Structure (15 sections, 15 equations, 9 figures, 1 table)

This paper contains 15 sections, 15 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Experiments
  5. How does model merging affect model performance?
  6. How do different merging methods affect final performance?
  7. How to determine the optimal interval and number of weights to merge for various model sizes?
  8. Do merged pre-trained models contribute to better downstream training?
  9. Does model merging improve the training stability?
  10. Investigating the Mechanisms of Model Merging
  11. Conclusion
  12. Acknowledgments
  13. The Effect of Model Merging in Dense Models
  14. Model Merging at the CT Stage for Supervised Fine-Tuning
  15. Limitations

Figures (9)

  • Figure 1: Comparison of downstream task performance for MoE models of varying sizes under stable training, before and after model merging.
  • Figure 2: Comparison of overall performance for MoE models of varying sizes under annealing training, before and after model merging. The learning rate follows a cosine schedule during the annealing process. The x-axis shows the count of training tokens.
  • Figure 3: Comparison of downstream task performance between model merging results under stable training and the real annealed model. The x-axis shows the count of training tokens.
  • Figure 4: Impact of different model merging methods on final model performance.
  • Figure 5: Impact of different model merging hyper-parameters on final model performance.
  • ...and 4 more figures