Training-Free Pretrained Model Merging

TL;DR

MuDSC introduces training-free model merging by enforcing dual-space unit similarity, blending activation- and weight-space representations through a linear combination controlled by $\alpha$ to guide permutation-based merging. The framework derives both activation-based and weight-space matching, adopts an iterative scheme to align units across layers, and extends to group-structured modules via group alignment and group zip. Empirical results across homogeneous and heterogeneous tasks show MuDSC consistently improves joint and per-task performance compared with state-of-the-art baselines, and loss-landscape analyses reveal merged models occupying overlapped, low-loss basins. The approach broadens practical applicability of training-free merging to diverse architectures and tasks, with public code available for reproducing results.

Abstract

Recently, model merging techniques have surfaced as a solution to combine multiple single-talent models into a single multi-talent model. However, previous endeavors in this field have either necessitated additional training or fine-tuning processes, or require that the models possess the same pre-trained initialization. In this work, we identify a common drawback in prior works w.r.t. the inconsistency of unit similarity in the weight space and the activation space. To address this inconsistency, we propose an innovative model merging framework, coined as merging under dual-space constraints (MuDSC). Specifically, instead of solely maximizing the objective of a single space, we advocate for the exploration of permutation matrices situated in a region with a unified high similarity in the dual space, achieved through the linear combination of activation and weight similarity matrices. In order to enhance usability, we have also incorporated adaptations for group structure, including Multi-Head Attention and Group Normalization. Comprehensive experimental comparisons demonstrate that MuDSC can significantly boost the performance of merged models with various task combinations and architectures. Furthermore, the visualization of the merged model within the multi-task loss landscape reveals that MuDSC enables the merged model to reside in the overlapping segment, featuring a unified lower loss for each task. Our code is publicly available at https://github.com/zju-vipa/training_free_model_merging.

Figures (3)

• Figure 1: We use an intermediate layer of ResNet50 resnet_he2016deep to construct two groups of units as two parents, and then merge them pairwise. In the figure, each point represents a merged unit. The x-axis and the y-axis denote the weight and activation similarity between their two parents respectively. The color indicates the smaller activation similarity between them and their two parents. The star depicts the best merged unit which is most activation-similar to its parents. More details are provided in the supplementary material.
• Figure 2: Left. Examples of vanilla structure and group structure.Right. An example of group alignment. First, we calculate the similarity between units. Next, we compute permutation and then calculate the average of matched similarity within each pairs of groups. Finally, we compute permutation for each pairs of groups and then set the permutation of unmatched pairs to zeros.
• Figure 3: The loss landscape visualization of three zip-based methods(MuDSC$_{Zip}$, Zipit and W. Zip). Stars, squares, and diamond mark the positions of the merged models, and black spots mark the positions of their parent models.