Transport and Merge: Cross-Architecture Merging for Large Language Models

TL;DR

This work tackles cross-architecture knowledge transfer by aligning intermediate activations of heterogeneous LLMs with an optimal-transport formulation. By deriving cross-model feature and layer correspondences, it enables targeted weight-space fusion through selective neuron replacement, using only a small calibration set and optional residual-frozen adaptation. The approach yields consistent improvements across multiple low-resource languages and expert domains, and proves robust to source backbone choices while providing a principled representation-space interpretation of weight transport. It offers a practical alternative to distillation for scenarios where architectures differ, with implications for rapid, data-efficient adaptation to new languages and domains.

Abstract

Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.

Figures (8)

• Figure 1: Cross-domain comparison using normalized scores (higher is better). Top: Domain-specific performance trajectories for Cantonese, Malaysian, and Thai under different transfer strategies. Bottom-left: Relative improvements within each domain. Bottom-right: Average normalized performance across all domains. See Section \ref{['sec:effect']} for detailed analysis.
• Figure 2: Illustration of cross-architecture merging pipeline. Given a small dataset $\mathcal{D}$, we extract intermediate activations from a high-resource source model and a low-resource target model with heterogeneous architectures. We then use optimal transport to infer layer- and feature-level correspondences, and leverage the resulting transport plans for direct parameter fusion. Finally, the fused model can be optionally refined via residual-frozen adaptation, where the transferred residuals are kept fixed and only base weights are updated.
• Figure 3: Average transport mass explained by the top-$k$ neuron correspondences, computed from the optimal transport plans and averaged over layers and modules.
• Figure 4: Sensitivity to the choice of source backbone. On the Malay and Indonesian benchmarks, our method consistently outperforms the base model when using either LLaMA3-8B or Qwen2.5-7B as the source model.
• Figure 5: Sensitivity analysis of the fusion coefficient $\alpha$ for our method (Fused w/ adaptation) on MalayMMLU. We report category-wise accuracy (%; higher is better); dashed lines denote the corresponding base-model performance.

Theorems & Definitions (4)

• Theorem 4.1: Representation-Space Interpretation of Weight Transport
• Corollary 4.2: Uniqueness Under Invertible Coordinate Maps
• proof : Proof of Theorem \ref{['thm:rep-transfer']}
• proof : Proof of Corollary \ref{['cor:invertible-alignment']}