DC-Merge: Improving Model Merging with Directional Consistency

TL;DR

DC-Merge, a method for directional-consistent model merging that first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented and achieves state-of-the-art performance in both full fine-tuning and LoRA settings.

Abstract

Model merging aims to integrate multiple task-adapted models into a unified model that preserves the knowledge of each task. In this paper, we identify that the key to this knowledge retention lies in maintaining the directional consistency of singular spaces between merged multi-task vector and individual task vectors. However, this consistency is frequently compromised by two issues: i) an imbalanced energy distribution within task vectors, where a small fraction of singular values dominate the total energy, leading to the neglect of semantically important but weaker components upon merging, and ii) the geometric inconsistency of task vectors in parameter space, which causes direct merging to distort their underlying directional geometry. To address these challenges, we propose DC-Merge, a method for directional-consistent model merging. It first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented. These energy-balanced vectors are then projected onto a shared orthogonal subspace to align their directional geometries with minimal reconstruction error. Finally, the aligned vectors are aggregated in the shared orthogonal subspace and projected back to the original parameter space. Extensive experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance in both full fine-tuning and LoRA settings. The implementation code is available at https://github.com/Tobeginwith/DC-Merge.

Figures (17)

• Figure 1: The singular value distribution of task vectors averaged across all layers. We normalize each singular value by the largest one within each dataset to eliminate the magnitude discrepancy among different datasets.
• Figure 2: (a) Comparison of cross-task transferability before (left) and after (right) balancing the energy distribution. The diagonal elements represent the relative performance of each task with respect to its fine-tuned model, while the off-diagonal elements indicate the relative transfer performance on other tasks, normalized against their zero-shot baseline. (b) Each task vector's cosine similarity with multi-task vector (lines) and average normalized transfer accuracy of each task vector (bars). We compare original task vectors and their energy-balanced counterparts. The energy-balanced task vectors achieve higher $\mathrm{CosSim}$ with the multi-task vector (we use $\boldsymbol{\Delta}_{\text{Iso-C}}$marczak2025notaskleftbehind for simplicity) and better cross-task generalization, indicating that balancing the energy distribution across knowledge components enhances multi-task expressiveness.
• Figure 3: (a) Comparison of $\mathrm{CosSim}$ and $\mathrm{DirSim}$. $\mathrm{DirSim}$ considers the similarity between every pair of directions equally, whereas $\mathrm{CosSim}$ mainly focuses on the similarity among the dominant directions while ignoring the minor ones. (b) Empirical validation of the importance of preserving directional geometry. Solid line: task performance vs. $\mathrm{DirSim}$ under random directional perturbations; dashed line: task performance vs. $\mathrm{CosSim}$ under energy distribution perturbations. (c) Correlation of task-wise performance with projected $\mathrm{DirSim}$. The overall correlation is positive (Pearson $R=0.64$) and per-method averages follow the same trend. Similar patterns persist under larger task scales as illustrated in Figure \ref{['fig_metric_boxplot']}. We utilize Normalized Accuracy Improvement (NAI) marczak2025notaskleftbehind to measure task-wise performance in (b) and (c).
• Figure 4: Optimization trajectory of the cover basis. We start from a poor initialization $\left(\widetilde{\boldsymbol{U}}_0, \widetilde{\boldsymbol{V}}_0\right)$ obtained by SVD of task vectors from TA and iteratively optimize the surrogate objective in Eq. \ref{['eq:sur-obj']}. (a) The optimization trace of alignment score and average normalized accuracy of DC-Merge under linear scale. (b) The same process in semi-log scale for better visibility of early-stage dynamics. Both alignment score and accuracy increase monotonically during training and converge to a stable optimum, demonstrating that higher alignment directly corresponds to stronger task retention. The final accuracy and alignment closely match those of the whitening-based approximation, confirming that whitening yields a near-optimal cover basis.
• Figure 5: The average absolute accuracy of DC-Merge with mask size $\{8r,4r,2r,r,r/2,r/4,r/8\}$ in FFT (left) and LoRA (right) settings. Performance peaks near $r$ for FFT and shows a milder variation for LoRA (slightly improves till $r$ then decreases), indicating that moderate masks retain performance while overly small/large masks degrade it. The results are based on ViT-B-32 8-task benchmark.

Theorems & Definitions (4)

• Proposition 1
• Proof 1
• Proposition 2
• Proof 2