Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration

TL;DR

The paper tackles the problem of merging multiple fine-tuned task-specific models by addressing feature drift as the primary source of negative transfer. It introduces Layer-wise Optimal Task Vector Merging (LOT Merging), a training-free method that casts merging as a layer-wise convex quadratic optimization to minimize feature drift, deriving closed-form solutions for linear weights, normalization scales, and biases. By analyzing extreme subspace configurations with SVD, the authors justify the layer-wise task-vector approach and demonstrate that it can adaptively preserve task-specific representations while minimizing interference, without retraining. Extensive experiments across vision and vision-language benchmarks show LOT Merging achieving state-of-the-art performance among training-free methods and robust performance with limited exemplars, including strong gains on ViT models and competitive results with vision-language models. The work provides practical, efficient, and scalable merging guidance and highlights future directions for handling complex layer operations and broader generalization.

Abstract

Multi-task model merging aims to consolidate knowledge from multiple fine-tuned task-specific experts into a unified model while minimizing performance degradation. Existing methods primarily approach this by minimizing differences between task-specific experts and the unified model, either from a parameter-level or a task-loss perspective. However, parameter-level methods exhibit a significant performance gap compared to the upper bound, while task-loss approaches entail costly secondary training procedures. In contrast, we observe that performance degradation closely correlates with feature drift, i.e., differences in feature representations of the same sample caused by model merging. Motivated by this observation, we propose Layer-wise Optimal Task Vector Merging (LOT Merging), a technique that explicitly minimizes feature drift between task-specific experts and the unified model in a layer-by-layer manner. LOT Merging can be formulated as a convex quadratic optimization problem, enabling us to analytically derive closed-form solutions for the parameters of linear and normalization layers. Consequently, LOT Merging achieves efficient model consolidation through basic matrix operations. Extensive experiments across vision and vision-language benchmarks demonstrate that LOT Merging significantly outperforms baseline methods, achieving improvements of up to 4.4% (ViT-B/32) over state-of-the-art approaches. The source code is available at https://github.com/SunWenJu123/model-merging.

Figures (3)

• Figure 1: An illustration of feature drift (measured as the cosine distance between features extracted by task-specific expert models and the merged model) is presented using Task Arithmetic task_arithmetic on eight vision tasks. (a) The feature drift in the last layer shows a linear correlation with the accuracy decrease, with data points collected for each task under varying merging coefficients. (b) Feature drift becomes more pronounced as the network depth increases.
• Figure 2: Performance (Avg ACC %) degradation of direct parameter merging compared to Individual under different numbers of exemplars when merging ViT-L/14 models. The error is decomposed into the impact of merging task vectors (red) and modifying pre-trained weights (orange).
• Figure 3: Average accuracy (%) on eight vision tasks with various numbers of exemplars (a) and $\lambda$ (scaling factor in Eq. \ref{['eq:final']}).

Theorems & Definitions (1)

• proof