AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse
You Lu, Jiyang Zhang, Bihuan Chen, Chaofeng Sha, Dingji Wang, Xin Peng
TL;DR
This paper addresses the challenge of reusing multiple task-specific models across domains without retraining. It shows that existing model merging techniques fail to generalize beyond LLMs due to heterogeneous architectures, and introduces AutoMerge, a block-aware, search-based framework that segments complex models and performs per-block merging via Bayesian optimization to maximize cross-task preservation while minimizing discrepancy. Across LLMs, image classification, and autonomous driving, AutoMerge achieves higher preservation rates and much lower preservation discrepancies compared with traditional methods, while delivering substantial reductions in training time and computational resources relative to full retraining. The work provides a practical pathway for robust model reuse in real-world, resource-constrained settings and lays groundwork for extending training-free merging to even more diverse architectures.
Abstract
Software reuse has long been recognized as a critical and widely studied topic in software engineering, offering substantial benefits in reducing development costs, improving software quality, and enhancing operational efficiency. This paradigm extends into deep learning through model reuse. Recently, model merging has emerged in the domain of large language models (LLMs) as a training-free approach that takes multiple task-specific models with the same architecture as source models and merges them without retraining, enhancing model reuse within LLMs. However, no prior work has systematically investigated whether such an approach can be effectively applied to other deep learning models with different architectures across domains. To bridge this gap, we present the first systematic study that evaluates five model merging techniques on three distinct model architectures across three domains: LLMs, image classification, and autonomous driving. Our findings reveal that directly applying existing model merging techniques leads to highly inconsistent results and falls notably short of their success within LLMs. Moreover, a single model merging technique often fails to handle the heterogeneous structural properties within a model, limiting its applicability to different model architectures across domains. Furthermore, the effectiveness of model merging techniques is highly sensitive to hyperparameter configurations, thereby constraining their potential for broader adoption. Inspired by these insights, we propose AutoMerge, a novel search-based model merging framework that first segments complex models into multiple heterogeneous blocks and then systematically explores the merging space to identify the merging technique and its hyperparameter configuration.