Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

TL;DR

This work tackles the persistent degradation in multi-task model merging by revealing that parameter conflicts suppress task-specific information even among similar tasks. It introduces DTS, a lightweight, approximation-based framework using SVD decomposition, four-group thresholding, and per-group scaling to preserve task-specific signals with only 1% extra storage per task. A data-free variant extends DTS to unseen tasks by weighting information via semantic similarity, enabling robust generalization without access to training data. Across vision, NLP, and generation backbones, DTS consistently surpasses state-of-the-art baselines and demonstrates strong unseen-task generalization, making it practical for memory-constrained deployments.

Abstract

Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, existing methods often experience substantial performance degradation compared with individually fine-tuned models, even on similar tasks, underscoring the need to preserve task-specific information. This paper proposes Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that preserves task-specific information with minimal storage overhead. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1\% additional storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.

Figures (4)

• Figure 1: Performance of the pairwise merged model: we pairwise merge models fine-tuned on SVHN netzer2011reading or CoLA warstadt2019neural with those fine-tuned on the other seven datasets, reporting the best and worst performance across these combinations.WA, RM, TA, TM, and DE refer to Weight-Averaging wortsman2022model, RegMean jin2022dataless, Task-Arithmetic ilharco2022editing, Ties-Merging yadav2023ties, and DARE yu2024language, respectively.
• Figure 2: Framework overview. In the approximation stage, we first apply singular value decomposition to the task-specific information. Next, a novel thresholding strategy is used to group the elements of each singular vector, followed by computing a scaling factor for each group. During inference, the task-specific information is reconstructed using the approximated singular vectors for each task.
• Figure 3: A toy example of our thresholding strategy. Thresholding provides a more fine-grained approximation by further partitioning positive and negative values into large/small groups.
• Figure 4: Performance (%) of merged model and additional memory requirement for different methods with RoBERTa as the backbone.