Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success

TL;DR

The paper challenges the notion of intrinsic mergeability by showing that compatibility between models depends on both the merging algorithm and partner tasks. It introduces an interpretable framework that linearly combines 28 pre-merge pairwise metrics to predict post-merge performance across four merging methods, validated with leave-one-task-out cross-validation on 20 tasks fine-tuned from CLIP ViT-B/16. Key findings reveal method-specific 'fingerprints' for mergeability, with subspace overlap and gradient alignment emerging as stable, method-agnostic prerequisites. These insights enable pre-screening and targeted fine-tuning strategies that promote gradient and subspace compatibility, offering a principled path toward merge-aware training and reliable multitask models.

Abstract

Model merging combines knowledge from separately fine-tuned models, yet success factors remain poorly understood. While recent work treats mergeability as an intrinsic property, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using linear optimization over a set of interpretable pairwise metrics (e.g., gradient L2 distance), we uncover properties correlating with post-merge performance across four merging methods. We find substantial variation in success drivers (46.7% metric overlap; 55.3% sign agreement), revealing method-specific "fingerprints". Crucially, however, subspace overlap and gradient alignment metrics consistently emerge as foundational, method-agnostic prerequisites for compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future fine-tuning strategies that explicitly encourage these properties.

Figures (5)

• Figure 1: Linear optimization procedure of metric coefficients to maximize the Pearson correlation with the true post-merge performance. Repeated for all methods, yielding per-method optimal coefficients. Green and orange depict the forward and backward flows, respectively.
• Figure 2: Comparison of metric category importance, normalized by the number of constituent metrics to account for varying category sizes. Left: Stacked bar chart showing the mean absolute coefficient magnitude per category for each merging method. Right: Grouped bar chart illustrating the relative importance of each category per method, demonstrating the proportional influence of a typical metric within each category.
• Figure 3: Per-method learned coefficients for all $28$ metrics, averaged across the $20$ leave-one-task-out folds.
• Figure 4: Per-method learned coefficients for all $28$ metrics, averaged across the $20$ leave-one-task-out folds.
• Figure 5: Left: Per-method learned coefficients for all $28$ metrics, average across the $20$ LOTO folds using $L_1$ penalty. Right: frequency of coefficients being non-zero.