Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent

TL;DR

This work tackles merging multiple fine-tuned models without access to their original data by reframing the problem as constrained optimization that minimizes the gap to each task model while preserving shared knowledge. It introduces adaptive projective gradient descent (doGe) with a data-free objective, a shared subspace to retain common representations, a modification vector Delta to resolve task conflicts, and task-aware, training-free lambda coefficients. Across vision and NLP benchmarks, doGe delivers state-of-the-art results, showing robust gains over prior data-free and TTA methods and good generalization to unseen tasks. The method is plug-and-play, scalable, and data-agnostic, enabling efficient multi-task model merging on diverse architectures.

Abstract

Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. Existing methods attempt to alleviate task conflicts by sparsifying task vectors or promoting orthogonality among them. However, they overlook the fundamental target of model merging: the merged model performs as closely as possible to task-specific models on respective tasks. We find these methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Based on our findings, we frame model merging as a constrained optimization problem ($\textit{i.e.}$, minimizing the gap between the merged model and individual models, subject to the constraint of retaining shared knowledge) and solve it via adaptive projective gradient descent. Specifically, we align the merged model with individual models by decomposing and reconstituting the loss function, alleviating conflicts through $\textit{data-free}$ optimization of task vectors. To retain shared knowledge, we optimize this objective by projecting gradients within a $\textit{shared subspace}$ spanning all tasks. Moreover, we view merging coefficients as adaptive learning rates and propose a task-aware, training-free strategy. Experiments show that our plug-and-play approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.

Figures (4)

• Figure 1: The effect of task numbers on average accuracy for ViT-B/32, with error bars representing the 95% confidence interval. As the number of tasks increases, negative transfer becomes more pronounced. Although our method initially performs lower than other methods, its performance decreases more slowly, demonstrating superior robustness when handling a larger number of tasks.
• Figure 2: (a) Cosine similarity matrices of task vectors for ViT-B/32. (b) A schematic representation of the subspace spanned by the task representations, depicted as a two-dimensional plane.
• Figure 3: An illustration of element magnitudes in the task vector, inspired by shen2024efficient. Best viewed when zoomed in.
• Figure 4: The average accuracy changes corresponding to different rank ratios in the subspace under ViT-B/32 architecture.