Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost

TL;DR

Param$\Delta$ presents a training-free fusion strategy that adds the weight difference $\Delta\Theta = \Theta_{\text{post}} - \Theta_{\text{base}}$ to an updated base checkpoint via $\Theta_{\text{Param}\Delta} = \Theta'_{\text{base}} + \Delta\Theta$, enabling post-training capabilities without data-intensive finetuning. It is validated across open-weight Llama3/3.1, Qwen, and DeepSeek-distilled models, achieving near-post-training performance and, in some cases, surpassing direct post-training on general and tool-use benchmarks. The authors propose four practical scenarios (general-purpose, task-specific, continual-pretraining, and multi-source fusion) and demonstrate robustness, transfer efficiency (near 1, $\gamma \approx 0.98$), and predictable scaling behavior across delta magnitudes. By reducing data and compute requirements and leveraging readily available checkpoints, Param$\Delta$ offers a cost-free pathway to keep open-weight models aligned with advances, accelerating iteration and democratizing access to state-of-the-art capabilities.

Abstract

The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it demands extensive high-quality data and poses risks like overfitting, alongside significant computational costs due to repeated post-training and evaluation after each base model update. This paper introduces $ParamΔ$, a novel method that streamlines post-training by transferring knowledge from an existing post-trained model to a newly updated base model with ZERO additional training. By computing the difference between post-trained model weights ($Θ_\text{post}$) and base model weights ($Θ_\text{base}$), and adding this to the updated base model ($Θ'_\text{base}$), we define $ParamΔ$ Model as: $Θ_{\text{Param}Δ} = Θ_\text{post} - Θ_\text{base} + Θ'_\text{base}$. This approach surprisingly equips the new base model with post-trained capabilities, achieving performance comparable to direct post-training. We did analysis on LLama3, Llama3.1, Qwen, and DeepSeek-distilled models. Results indicate $ParamΔ$ Model effectively replicates traditional post-training. For example, the $ParamΔ$ Model obtained from 70B Llama3-inst, Llama3-base, Llama3.1-base models attains approximately 95\% of Llama3.1-inst model's performance on average. $ParamΔ$ brings a new perspective on how to fully leverage models in the open-weight community, where checkpoints for base and instruct models are readily available and frequently updated, by providing a cost-free framework to accelerate the iterative cycle of model development.

Figures (9)

• Figure 1: Overall diagram of $\text{Param$\Delta$}{}$. Given a newly updated base model $\Theta_\mathrm{base}'$, we can obtain its post-trained version with no additional training cost, by simply adding the weight difference between the previous post-trained model $\Theta_\mathrm{post}$ and original base model $\Theta_\mathrm{base}$: $\Theta_\mathrm{post}' = \Theta_\mathrm{base}' + (\Theta_\mathrm{post} - \Theta_\mathrm{base}).$
• Figure 2: Cosine similarities of parameter differences from the feed-forward layers and attention layers from various post-trained Llama-series models and their corresponding base models.
• Figure 3: Weight norms distribution of parameter differences from the feed-forward layers and attention layers from various post-trained Llama-series models and their corresponding base models.
• Figure 4: Four representative scenarios to apply $\text{Param$\Delta$}{}$ for knowledge and capability transfer
• Figure 5: Relationship between the real performance of $\text{Param$\Delta$}{}$ models and their hypothetical performance. The high $R^2$ values suggests that the hypothetical performance is a reliable estimate of the actual performance of the $\text{Param$\Delta$}{}$ models