From Coefficients to Directions: Rethinking Model Merging with Directional Alignment

TL;DR

The paper reframes model merging as a problem of directional alignment, arguing that both parameter- and feature-space directions heavily influence merged performance. It introduces Merging with Directional Alignment (MDA), which uses simplex ETF geometry to align task directions in parameter space and employs rotation-based feature-space alignment with ETF targets. The approach combines low-rank SVD-based sharing, ETF projections, and Procrustes rotations, optimized via an entropy term, ETF alignment loss, and rotation regularization, and demonstrates improved performance and generalization across vision and NLP tasks. The work provides theoretical insights into generalization via Rademacher complexity reductions and empirical evidence of gains over state-of-the-art baselines, especially as task counts grow.

Abstract

Model merging has emerged as a practical paradigm for integrating multiple independently trained models into a single model without joint retraining. Previous studies have demonstrated the effectiveness of combining parameters through strategies such as parameter decomposition, coefficient optimization, and subspace learning, significantly reducing the need for expensive joint training and achieving strong empirical performance across diverse tasks. However, these approaches predominantly treat merging as a problem of parameter space decomposition or fusion coefficient optimization, while overlooking the critical role of directional information in both parameter and feature spaces. In practice, naïve merging introduces inconsistencies in dominant parameter directions and disrupts structural coherence across models, which can degrade performance. Moreover, coefficient-based optimization methods implicitly assume compatible feature-space directions across models. However, Neural Collapse indicates that class features follow structured directional patterns, which may differ across independently trained models, making coefficient optimization alone insufficient. In this work, we emphasize the importance of \emph{directional alignment} and introduce a unified geometric framework, \emph{Merging with Directional Alignment} (\method{}), which aligns directional structures consistently in both the parameter and feature spaces. Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.

Figures (5)

• Figure 1: Left Figure: A comparison between our method and previous methods. Our method constructs a simplex Equiangular Tight Frame (ETF) as a geometric basis to align task-specific parameter directions, thereby achieving directional consistency that better approximates the behavior of multi-class joint training. $\text{M1}, \text{N1}, \text{M2}$ and $\text{N2}$ are fusion coefficients, rather than model or classifier weights. Right Figure: Correlation between the directional deviation $\Delta_{\text{ETF}}$ (from the ideal ETF geometry) and the performance gap $\Delta_{\text{diff}}$ across tasks. Compared with task singular vectors (TSV) gargiulo2025task, our approach reveals a clear correlation between the directional deviation $\Delta_\text{etf}$ from the simplex ETF structure and the observed performance $\Delta_{\text{diff}}$ gap: larger deviations from the simplex ETF structure reliably indicate larger performance gaps across tasks.
• Figure 2: Ablation study of our method.
• Figure 3: Ablation on the coefficients of different modules in the loss function.
• Figure 4: Top Figure: Description of our approach for directional alignment in the parameter space. Bottom Figure: Description of our approach for directional alignment in the feature space.
• Figure : Parameter-Space Alignment

Theorems & Definitions (11)

• Definition 1: Simplex Equiangular Tight Frame
• Definition 2: Task Vector Fusion with Feature Direction Alignment
• Theorem 1: ETF Structural Coherence
• proof
• proof
• proof
• Theorem 2: Generalization bound
• proof
• Theorem 3: Rademacher Bound for Gram-Aligned Models
• Theorem 4: Rademacher Complexity Bound