Task Singular Vectors: Reducing Task Interference in Model Merging

TL;DR

This work shows that treating entire networks as flat parameter vectors overlooks crucial structure and induces task interference in model merging. By analyzing per-layer task matrices with SVD, it reveals a low-rank structure and defines Task Singular Vectors (TSV) to separate task-specific directions. The authors introduce TSV-Compress (TSV-C) for aggressive, loss-preserving compression and TSV-Merge (TSV-M) which decorrelates TSVs to reduce interference, achieving state-of-the-art merging performance with minimal hyperparameter tuning. Across ViT variants and 8/14/20-task benchmarks, TSV-M delivers substantial accuracy gains and TSV-C offers storage-efficient compression with negligible performance loss, highlighting a scalable path to multi-task models without extra training.

Abstract

Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.

Figures (18)

• Figure 1: Mean accuracy of a ViT-L-14 merged over 8, 14, and 20 tasks respectively. By significantly surpassing existing methods, TSV-M establishes the new state of the art in model merging.
• Figure 2: Mean absolute accuracy of the ViT-B-32 model across increasing fractions of retained singular components, averaged over 20 tasks. The red line represents the average accuracy of the original fine-tuned models with full-rank task matrices, while the green line shows the accuracies using low-rank approximations.
• Figure 3: Visualization of task interference among 8 tasks computed on the first attention layer of a ViT-B-32. The diagonal blocks display intra-task similarities, while the off-diagonal blocks illustrate inter-task similarities. The zoomed-in section highlights the interaction between the right singular vectors of the 3$^{\text{rd}}$ and 4$^{\text{th}}$ tasks.
• Figure 4: Absolute accuracy of a ViT-B-32 merged over 8, 14, and 20 tasks, respectively.
• Figure 5: Approximation error from the orthogonalization of the TSVs through Procrustes for the ViT-B-32 model across 8 tasks. The violin plots represent layer-wise approximation error distributions for $U$ and $V$ in both full-rank and low-rank cases.

Theorems & Definitions (9)

• Proposition 4.1
• proof
• Theorem 6.1
• Proposition C.1
• proof
• Corollary C.2
• proof
• Theorem 6.1
• proof