Weight Weaving: Parameter Pooling for Data-Free Model Merging

TL;DR

Weight Weaving presents a data-free, plug-and-play framework for model merging that pools parameter updates across a user-defined scaling-factor space using a pooling function. By operating on delta weights and augmented merges from any base merging method, it eliminates the need for privileged evaluation data while remaining compatible with existing SOTA approaches. Empirical results across vision multi-task learning, continual learning, and domain generalization show consistent gains, with average improvements up to 15.9 percentage points in data-free settings. The approach highlights the practical value of pooling diverse scaling factors and invites future work on smarter factor selection and pooling strategies to further enhance data-free model merging.

Abstract

Model merging provides a cost-effective and data-efficient combination of specialized deep neural networks through parameter integration. This technique leverages expert models across downstream tasks without requiring retraining. Most model merging approaches critically depend on scaling hyper-parameters $λ$, which weight each model's contribution globally or individually. Principled approaches for setting scaling factors without accessing any data (data-free) are scarce, often leading researchers to tune $λ$ using privileged data from the evaluation set, which is obviously unfeasible in practice. To address this limitation, we introduce Weight Weaving, a plug-and-play technique that pools model weights across $λ$ values search space using user-defined pooling functions, such as averaging, random selection, or even existing model merging methods. Our method demonstrates high modularity, imposing minimal constraints on the search space. It operates orthogonally to existing model merging methods and eliminates evaluation data requirements. We validate Weight Weaving across three ViT variants in three experimental setups: vision multi-task learning, vision continual learning, and domain generalization. Our method consistently improves the performance of several model merging methods, achieving average accuracy gains of up to 15.9 percentage points in a data-free setting.

Figures (3)

• Figure 1: Average accuracy (y-axis) when using privileged data is prohibited. Weight Weaving mostly enhances SOTA merging methods (x-axis) by large margins (Table \ref{['tab:table-ensemble']} for details). We consider three ViT model variants across multi-task learning, continual learning, and domain generalization experimental setups. Higher values indicate better performance.
• Figure 2: Distribution of optimal scaling factors across experimental setups for ViT-B-32. Each plot is a histogram where the x-axis represents the scaling factor value ($\lambda$), and the y-axis shows the number of times that value yields the best evaluation performance. Each column corresponds to a different experimental setup, while each row represents a model merging method: MagMax (top row) and ISO-C (bottom row). For MagMax, our method consistently improves performance, with optimal scaling factors spread across a range of values. In contrast, ISO-C's scaling factors are concentrated around $\lambda = 1$, in multi-task setting, suggesting that deviations from this value tend to reduce performance. Our method improves on continual learning and domain generalization setups, in which the optimal $\lambda$ values are broadly distributed without a clear central tendency. The blue line in each plot shows the Kernel Density Estimation over the histogram of optimal values.
• Figure 3: Pair-wise cosine similarity between task vectors for two fine-tuning strategies using the CIFAR-100 dataset split into 10 tasks. The left panel shows the results of sequential fine-tuning, where each task initializes from the weights of the previous one, leading to highly correlated task vectors (high cosine similarity). In contrast, the right panel shows individual fine-tuning, where each task initializes independently from the original pre-trained model. The latter results in low cosine similarity, indicating that the task vectors are substantially less correlated.