SeMe: Training-Free Language Model Merging via Semantic Alignment

TL;DR

SeMe tackles the challenge of combining strengths from multiple language models without retraining or access to original data. It introduces a semantic-alignment paradigm that treats LM semantics as vector-like components in latent space, enabling fine-grained, layer-wise merging while stabilizing internal knowledge. The approach relies on a three-step merging workflow (Select, Compute, Erase) and a suite of semantic techniques (vocabulary-defined semantics, semantic bases, and semantics-preservative transformations) to perform data-free fusion. Empirical and theoretical components suggest that SeMe can outperform data-dependent methods in both performance and efficiency, offering a scalable and interpretable path to knowledge-aware model composition.

Abstract

Despite the remarkable capabilities of Language Models (LMs) across diverse tasks, no single model consistently outperforms others, necessitating efficient methods to combine their strengths without expensive retraining. Existing model merging techniques, such as parameter averaging and task-guided fusion, often rely on data-dependent computations or fail to preserve internal knowledge, limiting their robustness and scalability. We introduce SeMe (Semantic-based Merging), a novel, data-free, and training-free approach that leverages latent semantic alignment to merge LMs at a fine-grained, layer-wise level. Unlike prior work, SeMe not only preserves model behaviors but also explicitly stabilizes internal knowledge, addressing a critical gap in LM fusion. Through extensive experiments across diverse architectures and tasks, we demonstrate that SeMe outperforms existing methods in both performance and efficiency while eliminating reliance on external data. Our work establishes a new paradigm for knowledge-aware model merging and provides insights into the semantic structure of LMs, paving the way for more scalable and interpretable model composition.

Figures (3)

• Figure 1: Semantic association of vocabulary and latent space. For each color label on the vocabulary (left), there is a color semantic basis in the latent space (middle). The semantics of the dark dot (indicating an arbitrary representation) in the latent space can be quantified as its cosine similarities to semantic bases. The semantics can be computed as probabilities on the vocabulary. When focusing on the nearest semantic basis for a given latent representation, a latent space can be quantified as discrete semantic regions (right).
• Figure 2: Empirical Validation of Semantics Decomposition (CodeGen).
• Figure 3: Semantic alignment solving the concerns of sequence segmentation and vocabulary mapping.