Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts

TL;DR

The paper tackles the lack of a unified theory for model merging under heterogeneous fine-tuning hyperparameters by introducing an $L_2$-stability based excess-error bound for the merged model $oldsymbol{x}_{avg}$. It unifies diverse merging strategies by showing they optimize different terms in the bound and provides actionable guidance on selecting hyperparameters to build merge-friendly experts. The empirical validation spans thousands of finetuning and merging trials on ResNet and ViT across 20 tasks, demonstrating strong alignment between theory and practice and offering practical insights for pretrain-to-merge pipelines. Overall, the work transforms model merging from an empirical craft into a principled framework with theory-driven recommendations and broad applicability to heterogeneous downstream tasks.

Abstract

Model merging efficiently aggregates capabilities from multiple fine-tuned models into a single one, operating purely in parameter space without original data or expensive re-computation. Despite empirical successes, a unified theory for its effectiveness under heterogeneous finetuning hyperparameters (e.g., varying learning rates, batch sizes) remains missing. Moreover, the lack of hyperparameter transparency in open-source fine-tuned models makes it difficult to predict merged-model performance, leaving practitioners without guidance on how to fine-tune merge-friendly experts. To address those two challenges, we employ $L_2$-Stability theory under heterogeneous hyperparameter environments to analyze the generalization of the merged model $\boldsymbol{x}_{avg}$. This pioneering analysis yields two key contributions: (i) \textit{A unified theoretical framework} is provided to explain existing merging algorithms, revealing how they optimize specific terms in our bound, thus offering a strong theoretical foundation for empirical observations. (ii) \textit{Actionable recommendations} are proposed for practitioners to strategically fine-tune expert models, enabling the construction of merge-friendly models within the pretraining-to-finetuning pipeline. Extensive experiments on the ResNet/Vit family across 20/8 visual classification tasks, involving thousands of finetuning models, robustly confirm the impact of different hyperparameters on the generalization of $\boldsymbol{x}_{avg}$ predicted by our theoretical results.

Figures (8)

• Figure 1: Empirical Validation of Theoretical Predictions in Transformer. The experiments evaluate four representative methods under different algorithmic hyperparameters in terms of average accuracy and average loss, both measured on $\boldsymbol{x}_{avg}$. Each subfigure also includes the theoretically predicted trend from Theorem \ref{['the:tight upper bound of excess error']}. As can be observed, all methods closely follow the theoretical predictions, demonstrating the applicability of our framework.
• Figure 2: Empirical Validation of Theoretical Predictions in ResNet. We analyze the impact of key finetuning hyperparameters on the performance of the merged model ($\boldsymbol{x}_{avg}$). Each subplot corresponds to varying one hyperparameter while keeping others fixed. Performance is measured by accuracy (higher is better) and loss (lower is better) on the joint test set, averaged over 15 random task groups. The observed trends robustly align with the trade-offs predicted by our $L_2$-Stability-based generalization error bound in Theorem \ref{['the:tight upper bound of excess error']}.
• Figure 3: Hyperparameter Impact on Model Merging Performance. Comprehensive analysis of four key finetuning hyperparameters across three ResNet architectures. Top row (a-d): accuracy vs. batch size, learning rate, training data ratio, and training steps. Bottom row (e-h): corresponding loss curves. Each experiment varies one hyperparameter while fixing others at defaults (batch size=256, lr=0.001, data ratio=1.0, steps=4000). Results averaged over 20 vision datasets with error bars showing standard error.
• Figure 4: Detailed results for the impact of fine-tuning steps ($K_i$). The top row (a, b) shows performance trends, while the bottom row (c, d) provides direct comparisons. The results exhibit a clear trade-off: performance initially improves with more steps but then degrades, particularly for ResNet-18 and ResNet-50. This inverted U-shaped trend for accuracy and U-shaped trend for loss strongly substantiates our theoretical prediction of a balance between optimization and generalization, as discussed in Section \ref{['sec:exp_k_i']}.
• Figure 5: Detailed results for the impact of batch size ($b_i$). The plots show that for all three backbones, increasing the batch size from 64 to 256 leads to an improvement in performance, as evidenced by decreasing average accuracy (a) and increasing average loss (b). This suggests that in our experimental setting, the potential benefits of reduced gradient variance—a key component of our theoretical bound—outweigh the negative impact on model stability from larger batches, as discussed in Section \ref{['sec:b_i']}.

Theorems & Definitions (20)

• Remark 1
• Definition 1
• Definition 2: $l_2$ on-average model stability
• Remark 2
• Remark 3
• Lemma 1: Generalization via on-average model stability
• Theorem 1
• Theorem 2
• Theorem 3
• Remark 4