Extrapolation Merging: Keep Improving With Extrapolation and Merging

TL;DR

This work tackles the data- and compute-heavy problem of aligning LLMs by introducing ExMe, a three-stage framework that combines model extrapolation with merging to improve performance without additional data or training. ExMe first derives two strong extrapolated directions from top SFT checkpoints and then merges them with a learned weighting $\beta$, guided by explicit extrapolation hyperparameters $\alpha$ and merging weight. The authors validate that extrapolation can provide directionally beneficial improvements during SFT and demonstrate that ExMe yields the strongest overall performance and robustness across seven benchmarks and multiple base models. The approach offers a practical pathway to more efficient model alignment with broad impact on real-world LLM deployment, while noting that formal proofs and redundancy-aware optimization remain open areas for further study.

Abstract

Large Language Models (LLMs) require instruction fine-tuning to perform different downstream tasks. However, the instruction fine-tuning phase still demands significant computational resources and labeled data, lacking a paradigm that can improve model performance without additional computational power and data. Model merging aims to enhance performance by combining the parameters of different models, but the lack of a clear optimization direction during the merging process does not always guarantee improved performance. In this paper, we attempt to provide a clear optimization direction for model merging. We first validate the effectiveness of the model extrapolation method during the instruction fine-tuning phase. Then, we propose Extrapolation Merging, a paradigm that can continue improving model performance without requiring extra computational resources or data. Using the extrapolation method, we provide a clear direction for model merging, achieving local optimization search, and consequently enhancing the merged model's performance. We conduct experiments on seven different tasks, and the results show that our method can consistently improve the model's performance after fine-tuning.

Figures (6)

• Figure 1: Overview of ExMe: ExMe process begins with a base model, from which SFT is conducted and the top two checkpoints with the best overall performance are selected. These SFT models are then extrapolated independently from the base model, resulting in two extrapolated models, EXPO-1 and EXPO-2. Finally, the two extrapolated models are directly weighted and merged, yielding the final merged model. In this figure, the color depth of the surface represents the model's performance, with lighter colors indicating stronger performance.
• Figure 2: Three stages of ExMe: In the fine-tuning stage, after the base model is fine-tuned, a series of checkpoints are generated, from which the two SFT models with the best overall performance are selected for the next stage of extrapolation. In the extrapolation stage, each of these SFT models emphasizes different capabilities, which are further amplified in the best two directions. In the merging stage, the merging process balances the strengths and mitigates the weaknesses of the two extrapolated models, resulting in improved overall performance.
• Figure 3: The extrapolation model of the gemma-2-2b-it model on HumanEval, MBPP, and GSM8K.
• Figure 4: The extrapolation model of the Llama3-8B model on HumanEval, MBPP, and GSM8K.
• Figure 5: This figure shows the impact of the extrapolation hyperparameter $\alpha$ on the performance of the extrapolated models across four different models, where each of the eight selected SFT models is extrapolated from the base model. The blue line represents the performance of the extrapolated models, while the red dashed line indicates the performance of the SFT models involved in the extrapolation.