Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer

TL;DR

SALT tackles cross-lingual transfer from English-centric LLMs to target-language LLMs by recycling target-language PLM embeddings. It constructs a per-token linear regression $X_{t_i}$ to map non-overlapping target vocabulary into the source embedding space, using semantically nearest shared tokens and a Sparsemax-based top-$k$ selection, with a final Language Adaptive Continual Pre-training step on target data. Across German, Arabic, and Vietnamese, SALT outperforms strong baselines on knowledge benchmarks and cross-lingual MRC, while converging faster and preserving English capabilities. The method is architecture-agnostic, extending to PLMs of encoder, decoder, and encoder-decoder types, indicating strong scalability for leveraging PLMs in modern LLM deployment.

Abstract

Large Language Models (LLMs) increasingly incorporate multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model's embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose Semantic Aware Linear Transfer (SALT), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies, to handle each non-overlapping token's embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods and achieves lower loss with accelerating faster convergence during language adaptation. Notably, SALT obtains remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures.

Figures (2)

• Figure 1: Summary of SALT. By using the paired embeddings of semantically similar tokens for each non-shared token, we create a unique least square matrix $X_{t_i}$ to transfer from PLM to LLM. $k$ denotes the number of selected nearest tokens among overlapping tokens for linear transfer, and $h_t$ and $h_s$ refer to the hidden dimensions of the target and source models.
• Figure 2: Causal Language Modeling (CLM) loss of language adaptive continual pre-training. The x-axis represents the progression of training steps, while the y-axis denotes the training loss. The first data point is logged at 20 steps.