Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging

Bang An, Yibo Yang, Philip Torr, Bernard Ghanem

TL;DR

This work tackles conflicts in task-vector-based model merging by purifying task updates in a knowledge-aware subspace. It uses context-oriented SVD (CO-SVD) to identify task-relevant components via covariance-driven projections, yielding purified vectors $ riangle W_{ ext{PAVE}} = ext{SVD}_r(W_{ ext{FT}}C)C^{-1} - W_{ ext{B}}$, and employs a spectral-rank allocation to balance contributions across tasks. Across GLUE benchmarks, generation tasks, and ViT architectures, PAVE consistently improves merging performance over baselines and random pruning, while remaining plug-and-play with multiple merging strategies. The approach demonstrates that knowledge-aware subspace decomposition paired with rank-aware pruning can effectively reduce task-irrelevant redundancy and conflicts in multi-task model merging, enabling more reliable and scalable assembly of specialized capabilities.

Abstract

Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training. In recent model merging methods, task vector has become a fundamental building block, as it can encapsulate the residual information from finetuning. However, the merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors. Existing efforts in overcoming redundancy by randomly dropping elements in the parameter space involves randomness and lacks knowledge awareness. To address these challenges, in this study, we propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace. Concretely, we sample some training examples from each task, and feed them into their corresponding fine-tuned models to acquire the covariance matrices before linear layers. We then perform a context-oriented singular value decomposition, which accentuates the weight components most relevant to the target knowledge. As a result, we can split fine-tuned model weights into task-relevant and redundant components in the knowledge-aware subspace, and purify the task vector by pruning the redundant components. To induce fair pruning efforts across models, we further introduce a spectral rank allocation strategy by optimizing a normalized activated pruning error. The task vector purification by our method as a plug-and-play scheme is applicable across various task vector-based merging methods to improve their performance. In experiments, we demonstrate the effectiveness of PAVE across a diverse set of merging methods, tasks, and model architectures.

Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging

TL;DR

This work tackles conflicts in task-vector-based model merging by purifying task updates in a knowledge-aware subspace. It uses context-oriented SVD (CO-SVD) to identify task-relevant components via covariance-driven projections, yielding purified vectors , and employs a spectral-rank allocation to balance contributions across tasks. Across GLUE benchmarks, generation tasks, and ViT architectures, PAVE consistently improves merging performance over baselines and random pruning, while remaining plug-and-play with multiple merging strategies. The approach demonstrates that knowledge-aware subspace decomposition paired with rank-aware pruning can effectively reduce task-irrelevant redundancy and conflicts in multi-task model merging, enabling more reliable and scalable assembly of specialized capabilities.

Abstract

Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training. In recent model merging methods, task vector has become a fundamental building block, as it can encapsulate the residual information from finetuning. However, the merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors. Existing efforts in overcoming redundancy by randomly dropping elements in the parameter space involves randomness and lacks knowledge awareness. To address these challenges, in this study, we propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace. Concretely, we sample some training examples from each task, and feed them into their corresponding fine-tuned models to acquire the covariance matrices before linear layers. We then perform a context-oriented singular value decomposition, which accentuates the weight components most relevant to the target knowledge. As a result, we can split fine-tuned model weights into task-relevant and redundant components in the knowledge-aware subspace, and purify the task vector by pruning the redundant components. To induce fair pruning efforts across models, we further introduce a spectral rank allocation strategy by optimizing a normalized activated pruning error. The task vector purification by our method as a plug-and-play scheme is applicable across various task vector-based merging methods to improve their performance. In experiments, we demonstrate the effectiveness of PAVE across a diverse set of merging methods, tasks, and model architectures.
Paper Structure (19 sections, 9 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 9 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Purifying task vectors in knowledge-aware subspace
  6. Spectral Rank Allocation
  7. Experiments
  8. Effect of decomposition in knowledge-aware subspaces
  9. Model merging for GLUE tasks
  10. Model Merging for generative tasks
  11. Ablation Studies
  12. Conclusion
  13. Implementation Details
  14. Model Merging on GLUE benchmark
  15. Model Merging on generation benchmarks
  16. ...and 4 more sections

Figures (4)

  • Figure 1: An illustration of our method and a comparison between PAVE and prior task vectors. DARE addresses redundancy by randomly dropping elements in the parameter space without knowledge awareness. PAVE refines task vectors by identifying the target knowledge-relevant components within knowledge-aware subspace and removing the redundant components.
  • Figure 2: An overall illustration of our proposed method, PAVE, which serves as a plug-and-play method for task-vector based merging methods. It begins by identifying task-aware components through a decomposition approach, and then eliminates noise and less effective components via rank-based pruning. A rank allocation strategy is designed to optimize the preserved rank ratios across individual models. Overall, rather than employing the plain task vector $W_{k} - W_{\text{B}}$, PAVE performs model merging based on purified task vectors $\text{SVD}_{r_k}(W_{k}C_k)C^{-1}_k - W_{\text{B}},\ 1\le k \le K$.
  • Figure 3: Model rank pruning with different decomposition methods. The results show that the context-oriented decomposition (CO-SVD) is more effective in aligning the subspace with the target knowledge. The pruned ranks, 96, 192, 384 and 512, correspond to removing 1/8, 1/4, 1/2 and 2/3 of the full rank, respectively.
  • Figure 4: Performance and runtime analysis of PAVE across different sample sizes. Left: Mean performance (solid line) and standard deviation (shaded region) of EMR-Merging with PAVE, showing consistent improvements as more samples are used. Right: PAVE runtime grows with number of samples.