Fine-Grained Model Merging via Modular Expert Recombination

TL;DR

MERGE tackles the inefficiency and limited reuse of instance-specific model merging by introducing a fine-grained, component-wise merging framework. It offline-optimizes a diverse set of Pareto-optimal merging configurations using a surrogate-assisted bi-objective evolutionary search to build a reusable modular-expert library, and then enables input-aware on-demand recombination at inference via a lightweight routing network. The approach reveals systematic component heterogeneity and provides practical gains across vision, language, and PEFT settings, including substantial storage reductions while preserving or improving task performance. By decoupling offline optimization from online deployment and leveraging modular experts, MERGE enables scalable, batch-friendly, and resource-aware multi-task deployment with broad applicability and robust generalization.

Abstract

Model merging constructs versatile models by integrating task-specific models without requiring labeled data or expensive joint retraining. Although recent methods improve adaptability to heterogeneous tasks by generating customized merged models for each instance, they face two critical limitations. First, the instance-specific merged models lack reusability, restricting the exploitation of high-quality merging configurations and efficient batch inference. Second, these methods treat each task-specific model as a monolithic whole, overlooking the diverse mergeability of homologous components such as attention and multilayer perceptron layers, and the differing merging sensitivities across components. To address these limitations, we propose MERGE (\underline{M}odular \underline{E}xpert \underline{R}ecombination for fine-\underline{G}rained m\underline{E}rging), a method that enables component-wise model merging and input-aware, on-demand module recombination at inference. MERGE formulates component-wise merging as a bi-objective optimization problem that balances cross-task performance and storage efficiency, and develops a surrogate-assisted evolutionary algorithm to efficiently identify Pareto-optimal merging configurations. These high-quality configurations underpin a reusable modular expert library, from which a lightweight routing network dynamically activates and recombines modular experts to assemble input-specific models and enable efficient inference under storage constraints. Extensive experiments across various model scales, task types, and fine-tuning strategies demonstrate that MERGE consistently outperforms strong baselines and generalizes effectively.

Figures (8)

• Figure 1: Exploratory analyses underscore the necessity of fine-grained merging. (a) Existing methods that generate input-agnostic models exhibit substantial performance degradation compared to task-specific models, highlighting the importance of input-aware merging. Darker colors indicate greater degradation. (b) Component-wise comparisons reveal heterogeneous merging sensitivities across components, emphasizing the need for component-wise merging.
• Figure 2: Overview of MERGE. (a) MERGE jointly achieves component-wise flexibility and input-aware adaptability in fine-grained merging. Component-wise merging obtains high-quality, reusable modular experts through offline merging of homologous components, while input-aware module recombination adapts merged models to various inputs during inference. (b) MERGE workflow. During offline optimization, MERGE introduces surrogate-assisted bi-objective evolutionary optimization to identify Pareto-optimal component-wise merging configurations that balance performance and storage cost. These configurations underpin the construction of a reusable modular expert library. During on-demand reuse at inference, guided by a lightweight routing network, MERGE recombines modular experts from this library to assemble input-specified models under user-specified resource constraints.
• Figure 3: Comparison of Pareto-optimal solutions obtained by different methods across various models. (a) Merging ViT-B/32 models. (b) Merging ViT-L/14 models. (c) Merging RoBERTa models. (d) Merging GPT-2 models. (e) Merging PEFT models.
• Figure 4: Merging utility of MERGE on various models. (a) ViT-B/32 models. (b) ViT-L/14 models. (c) RoBERTa models. (d) GPT-2 models. (e) PEFT models.
• Figure 5: Relation networks of various models by components. (a) ViT-B/32 models on 8 tasks. (b) ViT-L/14 models on 8 tasks. (c) RoBERTa models on 8 tasks. (d) GPT-2 models on 7 tasks. (e) PEFT models on 11 tasks.