Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models

TL;DR

Sens-Merging introduces a sensitivity-guided framework to adjust merging coefficients for task-vector-based merging, operating at both within-task layer importance and cross-task transferability. By combining layer-wise parameter sensitivity $oldsymbol{ ext{alpha}}^l$ with cross-task alignment $ au$, and fusing them through a softmax with temperature $T$, it yields per-layer coefficients $oldsymbol{\sigma}^l$ that enhance merged models when integrated with baselines like DARE. Empirical results across Mistral 7B and LLaMA2-7B/13B show consistent gains across general knowledge, mathematical reasoning, and especially code generation tasks, sometimes surpassing specialized fine-tuned models. The work also reveals trade-offs between task-specific and cross-task scalings and demonstrates scalability to larger models, while acknowledging limitations for heterogeneous architectures and larger weight divergences.

Abstract

Recent advances in large language models have led to numerous task-specialized fine-tuned variants, creating a need for efficient model merging techniques that preserve specialized capabilities while avoiding costly retraining. While existing task vector-based merging methods show promise, they typically apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. We present Sens-Merging, a sensitivity-guided coefficient adjustment method that enhances existing model merging techniques by operating at both task-specific and cross-task levels. Our method analyzes parameter sensitivity within individual tasks and evaluates cross-task transferability to determine optimal merging coefficients. Extensive experiments on Mistral 7B and LLaMA2-7B/13B models demonstrate that Sens-Merging significantly improves performance across general knowledge, mathematical reasoning, and code generation tasks. Notably, when combined with existing merging techniques, our method enables merged models to outperform specialized fine-tuned models, particularly in code generation tasks. Our findings reveal important trade-offs between task-specific and cross-task scalings, providing insights for future model merging strategies.

Figures (3)

• Figure 1: Sens-Merging functions as a plug-and-play enhancement to existing task vector-based merging techniques. Notably, when integrated with DARE, it surpasses even specialized code models in code generation.
• Figure 2: Overall framework of our Sens-Merging method. Sens-Merging adjusts layer-wise scaling coefficients for task-specialized fine-tuned models through two mechanisms: task-specific scaling and cross-task scaling.
• Figure 3: Layer-wise sensitivity scores across different task-specific models, with the Top-5 most sensitive layers highlighted in red.