MAGIC: Achieving Superior Model Merging via Magnitude Calibration

TL;DR

The paper identifies magnitude deviation as a key, previously underexplored factor in model merging, and introduces MAGIC, a plug-and-play framework with Feature Space Calibration, Weight Space Calibration, and Dual Space Calibration. By performing layer-wise, data-efficient calibration that accounts for magnitude-sensitive layers, MAGIC achieves significant performance gains across computer vision and natural language tasks without additional training. The approach combines theoretical analysis with practical, training-free methods, including data-driven FSC and data-free WSC/DSC, and demonstrates robustness across backbones and merging baselines. This work advances model merging by addressing magnitude as a core dimension, enabling more stable and scalable multi-task merging in real-world settings.

Abstract

The proliferation of pre-trained models has given rise to a wide array of specialised, fine-tuned models. Model merging aims to merge the distinct capabilities of these specialised models into a unified model, requiring minimal or even no additional training. A core objective of model merging is to ensure the merged model retains the behavioural characteristics of the specialised models, typically achieved through feature alignment. We identify that features consist of two critical components: direction and magnitude. Prior research has predominantly focused on directional alignment, while the influence of magnitude remains largely neglected, despite its pronounced vulnerability to perturbations introduced by common merging operations (e.g., parameter fusion and sparsification). Such perturbations to magnitude inevitably lead to feature deviations in the merged model from the specialised models, resulting in subsequent performance degradation. To address this, we propose MAGnItude Calibration (MAGIC), a plug-and-play framework that rectifies layer-wise magnitudes in feature and weight spaces, with three variants. Specifically, our Feature Space Calibration (FSC) realigns the merged model's features using a small set of unlabelled data, while Weight Space Calibration (WSC) extends this calibration to the weight space without requiring additional data. Combining these yields Dual Space Calibration (DSC). Comprehensive experiments demonstrate that MAGIC consistently boosts performance across diverse Computer Vision tasks (+4.3% on eight datasets) and NLP tasks (+8.0% on Llama) without additional training. Our code is available at: https://github.com/lyymuwu/MAGIC

Figures (11)

• Figure 1: Current merging methods seek to approximate specialised models, whose features serve as the optimal reference. While features consist of both direction and magnitude, existing approaches overlook magnitude alignment. Our calibration method improves performance consistently by aligning feature magnitudes more closely with the optimal.
• Figure 2: An illustration of task vectors based model merging methods. (a) Definition of task vectors, where $\theta_\text{pre}$ is the parameter of a pre-trained model, and $\theta_k$ is the model's weight fine-tuned on the $k$-th task. (b) Task Arithmetic Ilharco_2022_Editinga merges models via the arithmetic mean of task vectors. (c) Layer-wise AdaMerging Yang_2024_AdaMerging uses unlabelled data to learn different merge coefficients $\lambda_k^l$ for the task vector of each task $k$ and layer $l$. (d) Our method applies post-calibration after model merging, using adaptive estimation calibration coefficients $\xi^l$ to adjust both the task vector $\Delta \theta_\mathrm{merge}^l$ and task feature $\Delta h_{\mathrm{merge}}^{l}(x_k)$ at each layer.
• Figure 3: Influence of magnitude-altering operations (Left: TRIM, Middle: Arithmetic, Right: Disjoint) on the magnitude of weight and feature spaces. Each point in the figure represents the ratio between the modified magnitude after the operation and the original magnitude before the operation. So, in most cases, variations in feature magnitude closely follow those in weight magnitude, while certain fusion scenarios (c) reveal distinct patterns of variation.
• Figure 4: Visualisation of weight disentanglement property: given a merged model, when the task vector of a specific layer $l$ is replaced with that from the specialised model corresponding to the inference task, the feature magnitude of the output at that layer increases. The data in the $i$-th row and $j$-th column represents the increase in the task feature, measured by $L_1$ norm, resulting from substituting $\Delta \theta_j^l$ for $\Delta \theta_\textrm{merge}^l$ during inference on task $i$.
• Figure 5: Visualisation of the layer sensitivity. Different layers exhibit distinct sensitivity, while the same layers across different task vectors behave similarly. So, a universal task (e.g., CIFAR100) can be utilised to approximate the magnitude-sensitive layers in our target tasks (e.g., DTD or Cars).

Theorems & Definitions (4)

• Proposition 1: Norm Instability under Fusion
• Lemma 1: Task Feature Magnitude
• Theorem 1: Layer-wise Sensitivity to Task Vector Scaling
• Theorem 2: Optimal Layer-wise Scaling