M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Data
Tiantong Wang, Yiyang Duan, Haoyu Chen, Tiantong Wu, Wei Yang Bryan Lim
TL;DR
The paper tackles the data- and compute-efficient consolidation of multiple pretrained models by introducing M-Loss, a metric that quantifies how closely parameter averaging approximates ensembling using limited unlabeled data. It defines layer- and node-level M-Loss based on intermediate representations, derives expected values for common activations, and provides theoretical justification for mergeability when source models come from a shared pretrained backbone. Building on this, the authors propose M-TIES, a dynamic pruning-based merging method that uses M-Loss to allocate pruning budgets and reduce parameter conflicts, achieving superior merge performance with limited data and modest computation. Empirical evaluation on ViT backbones across diverse datasets demonstrates strong mergeability, competitive accuracy relative to ensembling, robustness to sampling randomness, and practical efficiency, with extensions like Few-Layer M-TIES offering further speedups. Overall, the work delivers both a solid theoretical lens and a practical toolkit for scalable model consolidation in low-data regimes.
Abstract
Training of large-scale models is both computationally intensive and often constrained by the availability of labeled data. Model merging offers a compelling alternative by directly integrating the weights of multiple source models without requiring additional data or extensive training. However, conventional model merging techniques, such as parameter averaging, often suffer from the unintended combination of non-generalizable features, especially when source models exhibit significant weight disparities. Comparatively, model ensembling generally provides more stable and superior performance that aggregates multiple models by averaging outputs. However, it incurs higher inference costs and increased storage requirements. While previous studies experimentally showed the similarities between model merging and ensembling, theoretical evidence and evaluation metrics remain lacking. To address this gap, we introduce Merging-ensembling loss (M-Loss), a novel evaluation metric that quantifies the compatibility of merging source models using very limited unlabeled data. By measuring the discrepancy between parameter averaging and model ensembling at layer and node levels, M-Loss facilitates more effective merging strategies. Specifically, M-Loss serves both as a quantitative criterion of the theoretical feasibility of model merging, and a guide for parameter significance in model pruning. Our theoretical analysis and empirical evaluations demonstrate that incorporating M-Loss into the merging process significantly improves the alignment between merged models and model ensembling, providing a scalable and efficient framework for accurate model consolidation.